Simplecov: 0.19.0 change of `Result.from_hash` broke our coverage merging

[mobilutz]

NOTE: This is NOT an issue, that breaks our implementation anymore.
This ticket is more created to talk about this topic as mentioned here as well: https://github.com/simplecov-ruby/simplecov/pull/913#issuecomment-677437282

The behaviour change in v0.19.0 for Result.from_hash lead to some problem on the way we merge our ruby coverages.

This is the PR that for us was an unexpected change, specifically the change that the method now returns an array instead of a Result object.
https://github.com/simplecov-ruby/simplecov/pull/913

Maybe it is good, to describe how we use Result.from_hash.
This is the code, that broke for us:

results = []
all_results = Dir['/tmp/coverage/.resultset*.json']
all_results.each do |result_file_name|
  puts "Processing #{result_file_name}"
  results << SimpleCov::Result.from_hash(JSON.parse(File.read(result_file_name)))
end

SimpleCov::ResultMerger.merge_results(*results).format!

What exactly does this code do?

We are running our ruby suite on 8 different processes which produce 8 different .resultset.json files. After all of them are created, we combine them with the code above.
This code was written almost 2 years ago, and worked without any problems. I am not sure anymore, what other ways of merging the different results has been tried back then. Even though I worked on that, I cannot remember. The only thing I do remember is, that this code lead to a complete HTML coverage report generation, and the others did not.

Maybe there is an easier way to combine results from different runs, that do not share the same storage possibilities except S3.

PS: We now "solved" this situation by replacing the << inside the code above to +=. So it was not too hard for us to bring back our functionality - the hard part was finding what needed to be changed!

[PragTob]

👋

so what you do is eerily similar to what we do in collate and also to how I changed it. I'd recommend you to do a map, or just start using our collate: https://github.com/simplecov-ruby/simplecov/pull/913/files#diff-107ed4983c51dd50042b98843dbaad97R85-R94

collate docs: https://github.com/simplecov-ruby/simplecov#merging-test-runs-under-different-execution-environments

How are you running the 8 processes? With parallel_tests then it should work without any changes.

I believe even if you don't use parallel tests and just run 8 test processes with appropriately different command_name's and a big merge_timeout it should be merged correctly (- existing race conditions): https://github.com/simplecov-ruby/simplecov#merging-test-runs-under-the-same-execution-environment

Cheers,
Tobi

[mobilutz]

We are using HerokuCI as our CI, and there the 8 independent "containers" are running in total separation. We use knapsack to distribute the specs to the so called nodes, and inside every node, we copy the .resultset to an S3 bucket.
When the bucket (folder) has 8 .resultset.json files, we merge them together.

I already tried to use collate, but when I run it like in the linked suggestion, the coverage-html that comes out of it is empty. But maybe I am just missing some configuration/setup.
All the resultsets have a uniq name, so that is node the problem. Example:

{
  "NODE 1 of 8": {
    "coverage": { [...] },
    "timestamp": 1597750265
  }
}

[PragTob]

@mobilutz that is very weird... it's made exactly for that use case so either it's a bug or some thing required that we haven't documented as good as it could be. Off the top off my head:

• were you're trying to merge, does it have access to the source code files under the path as shown in the coverage data?

hm no that's the only thing I can come up with... iirc it doesn't look at timeout times.

If you did a bit of debugging where it went wrong that'd be great but I know it takes time and you have a working solution. I'm not sure I can be of help, as I'd need the source code to fully see it work. theoretically a sample repo with 2 resultset jsons and some dummy files were collate doesn't work would be enough though.

I'm also gonna @deivid-rodriguez as he did a lot of the collating :+1: :grin:

[deivid-rodriguez]

Yeah, since this is the exact usecase of SimpleCov.collate, it'd be nice to figure this out :+1:

[mobilutz]

@PragTob I will try to look into it a bit more. And also create a small repo where it can be replicated with.

Does that repo need to have some *.rb source code as well? Or is it enough to just add some .resultset.json files with some dummy content?

[sterankin]

I am going to jump in here with the same issue.

I use Bamboo and as such I run 4 different jobs, each running a different set of RSpec tests on the same codebase (using different rake tasks).
Group 1 would run a specific set of rspec tests (e.g. model and services specs)
Group 2 would run a specific set of rspec tests (e.g. controller specs)
etc

I therefore create 4 .resultset.json files (one from each group).

In Bamboo, after all the tests have completed, I pull these 4 .resultset.json files into 4 different folders, i.e.:

group1/.resultset.json
group2/.resultset.json
group3/.resultset.json
group4/.resultset.json

I then run SimpleCov collate using a rake task, which generates output in the coverage folder

SimpleCov.collate Dir["group*/*resultset.json"], 'rails'

However, the resulting .resultset.json file in the coverage folder looks like this:

{
  "RSpec, RSpec, RSpec, RSpec": {
    "coverage": {
    },
    "timestamp": 1598005200
  }
}

I have verified this behaviour locally too.

The initial .resultset.json files from the 4 rspec groups seem to cover every file in the codebase, i.e. they are all the same except for the lines parts.

e.g. here is a snippet of the group1 .resultset.json file:

{
  "RSpec": {
    "coverage": {
      "/usr/src/app/lib/core_ext/array.rb": {
        "lines": [
          1,
          1,
          2,
          2,
          null,
          0,
          null,
          null,
          null,
          null
        ]
      },

And here is part of the group 2 .resultset.json file:

{
  "RSpec": {
    "coverage": {
      "/usr/src/app/lib/core_ext/array.rb": {
        "lines": [
          1,
          1,
          0,
          0,
          null,
          0,
          null,
          null,
          null,
          null
        ]
      },

As you can see, the files contain the same ruby files, but the lines sections are different.
Could I be missing some setting which would cause this?

[deivid-rodriguez]

Does that repo need to have some *.rb source code as well? Or is it enough to just add some .resultset.json files with some dummy content?

Ideally it includes everything, including a list of commands in the README that we can run after cloning the repository and see the issue for ourselves.

@sterankin Your issue sounds like the same problem to me.

[PragTob]

Yeah having .rb files is helpful because I worry that the missing .rb files is what causes the error. Using the example of "/usr/src/app/lib/core_ext/array.rb" - if there's no code at that point on the machine where you call collate then simplecov will skip those files.

Ofc with a different structure on the CI than locally that sucks and we might want to introduce either relativ paths.. overriding paths or something to make that workable.

[sterankin]

In my case, a couple of things to note.

In results_merger.rb method: def merge_results(*results)

the following line:
result.command_name = results.map(&:command_name).sort.join(", ")
produces:
"RSpec, RSpec, RSpec, RSpec"

I changed the code to:
result.command_name = results.map(&:command_name).uniq.sort.join(", ")

which results in:
"Rspec"

I am not sure if that is an issue however.

The missing output seems to be caused in the following method:

def store_result(result)

This following line produces missing data (even though the result object is populated).

command_name, data = result.to_hash.first

Please see attached screenshots for evaluation of these variables:

Before the line command_name, data = result.to_hash.first, the object result looks like:

After line command_name, data = result.to_hash.first, we can see the following:

[sterankin]

Edit: I don't think this is correct, its missing the "Coverage" section in the resulting .resultset.json

The following seems to fix it for me, need to test it further:

      def merge_results(*results)
        parsed_results = JSON.parse(JSON.dump(results.map(&:original_result)))
        combined_result = SimpleCov::Combine::ResultsCombiner.combine(*parsed_results)
        result = SimpleCov::Result.new(combined_result)
        # Specify the command name
        result.command_name = results.map(&:command_name).uniq.sort.join(", ") # added uniq here
        result
      end

      # Saves the given SimpleCov::Result in the resultset cache
      def store_result(result)
        synchronize_resultset do
          # Ensure we have the latest, in case it was already cached
          clear_resultset
          new_set = resultset
# start of changes
          command_name = result.command_name
          data = result.original_result
# end of changes
          new_set[command_name] = data
          File.open(resultset_path, "w+") do |f_|
            f_.puts JSON.pretty_generate(new_set)
          end
        end
        true
      end

[PragTob]

@sterankin I believe in your case the problem is that all the command names are the same (all rspec), please check the section about test suite names: https://github.com/simplecov-ruby/simplecov#test-suite-names

[sterankin]

@sterankin I believe in your case the problem is that all the command names are the same (all rspec), please check the section about test suite names: https://github.com/simplecov-ruby/simplecov#test-suite-names

Thanks - I think you could be right. I changed each test group to have a unique name through the environment variable.
However I am seeing the following result:

{
  "Job RSpec:GroupA, Job RSpec:GroupB, Job RSpec:GroupC, Job RSpec:GroupD": {
    "coverage": {
    },
    "timestamp": 1598350485
  }
}

I think this is an issue because of the file paths. Our RSpec tests run in Docker containers, and therefore the .resultset.json file has the following paths:

 "coverage": {
      "/usr/src/app/lib/core_ext/array.rb":

However once I try and collate all the .resultset.json files, the path of usr/src does not exist and it cannot find the code files - does this mean Simplecov will not output results?

[PragTob]

@sterankin yup absolutely right, simplecov needs the files to process things like disabling coverage or well display them in the HTML formatter etc... we should build a solution for that as I think it's rather common... like.. I dunno... either paths relative from the root directory or rewrite paths or something,

[sterankin]

@sterankin yup absolutely right, simplecov needs the files to process things like disabling coverage or well display them in the HTML formatter etc... we should build a solution for that as I think it's rather common... like.. I dunno... either paths relative from the root directory or rewrite paths or something,

I solved my issue by also running the SimpleCov collate in Docker also. However I have just found out that Sonar does not understand the new .resultset.json file introduced in Simplecov 0.18:

https://community.sonarsource.com/t/ruby-coverage-simplecov-json-file-does-not-work/20909/6

I can't go back to 0.17 since I need the collate functionality. Any ideas - does the json structure differ, could I monkey patch something?

[deivid-rodriguez]

Sonar should add support for recent simplecov versions. The discussion in codeclimate's test-reporter's repository probably applies to sonar in a similar way. To circumvent that and and upgrade now, you can convert the resultset file manually to the old format. Here's a script to do it: https://gist.github.com/qortex/7e7c49f3731391a91ee898336183acef#file-transform-rb.

[mobilutz]

@sterankin yup absolutely right, simplecov needs the files to process things like disabling coverage or well display them in the HTML formatter etc... we should build a solution for that as I think it's rather common... like.. I dunno... either paths relative from the root directory or rewrite paths or something,

@PragTob This is actually the situation that I didn't know. SimpleCov is expecting the results to have absolute paths.

I tried collate locally, without changing the paths in the .resultset.json files which of course then meant to output at all. I compared that to our working (fixed) solution on CI, which resulted in a correct coverage report.

After I created this repo, which I would have hoped to show the problem between collate and our custom-merger, I saw that the problem was "just" the incorrect absolute paths.

Maybe some information on this could be added.

PS: Here is the repo I created: https://github.com/mobilutz/simplecov-issue-920
NOTE: to really test the coverage merge/collate, one needs to change the path inside the two .resultset.json files ;)

[PragTob]

@mobilutz thanks!

Yeah absolutely I think what needs to be done here is the following:

1. document this current limitation better
2. provide an easy way to circumvent this limitation, I guess absolute paths were introduced/used for a reason. So not sure we could just fall back to relative paths... might require a bit of digging though. Alternatively, optionally enabling relative paths or some sort of path replace mechanism might be enough of a solution.

I'd probably move that to another issue though... there's actually loads of issues I should still create :|

Might take some time tomorrow!

[FGoessler]

We ran into the same issue with our setup on CircleCI and were able to fix it for now using the changes initially mentioned by @mobilutz on our merging code. Thanks for pointing us there!

FYI: We based our stuff off this gist: https://gist.github.com/trev/9cc964a54c8d5b62f4def891eba6b976

[PragTob]

@FGoessler is there anything keeping you from using straight up SimpleCov.collate ? That's what I'd recommend everyone to switch to :D

[mobilutz]

I will close this ticket now, as the problem is purely on our side.

Nevertheless would the followup tickets be very good to have @PragTob.
Thanks again for your feedback!