Simplecov: No report generated using RSpec Rails / SimpleCov 0.18.3

[odlp]

Hello,

First of all thank-you to everyone for all the hard work making the 0.18 release!

I think I've encountered a bug where no report is generated at the end of the test suite. This is in a Rails application tested using RSpec (using the rspec-rails gem), with SimpleCov 0.18.3.

Reproduction

I've made a minimal reproduction of the issue here:
https://github.com/odlp/simplecov-at-exit-bug

Tracing the cause

When ActiveSupport::TestCase is required, by RSpec Rails for example:

https://github.com/rspec/rspec-rails/blob/e23efbb2cd6b40aaa8c6f33a36fe7ed4e724e2c7/spec/rspec/rails/matchers/redirect_to_spec.rb#L2

This ends up requiring minitest:

https://github.com/rails/rails/blob/c70112e74f4d2ef517f4036fe6e2888cc30fc952/activesupport/lib/active_support/test_case.rb#L4

Unfortunately Minitest is a dependency of ActiveSupport:

https://github.com/rails/rails/blob/c70112e74f4d2ef517f4036fe6e2888cc30fc952/activesupport/activesupport.gemspec#L38

By having defined Minitest, this means the SimpleCov at_exit hook will skip
the 'at exit behaviour' hook:

https://github.com/colszowka/simplecov/blob/a179ec6dc419c43bce472c2426f30f24cc49b42f/lib/simplecov/defaults.rb#L24-L29

I've verified this by placing a breakpoint in the at_exit hook and stepping over the execution.

PRs which introduced this behaviour:

• https://github.com/colszowka/simplecov/pull/855
• https://github.com/colszowka/simplecov/pull/756

Temporary Workaround

RSpec.configure do |config|
  config.after(:suite) { SimpleCov.at_exit_behavior }
end

Or use versions '<= 0.18.2'

[odlp]

Forgot to mention I'm happy to contribute a PR to fix this, but would appreciate confirmation of the issue & any guidance on any preferred direction of the solution.

My initial thoughts for a fix would be to define a after(:suite) callback to trigger SimpleCov.at_exit_behavior if RSpec is defined, similar to the Minitest hook. Unfortunately if both RSpec and Minitest are defined I don't believe we can tell which is actually running (I haven't spotted a global flag or such in either library).

[nathanvda]

I can confirm, just started a new rails 6 project with rspec and simplecov did not generate a report, and adding the (temporary) workaround mentioned by @odlp works for me :clap: :clap:

[PragTob]

:wave:

Hi @odlp and thank you so much for your bug report and your in depth investigation tracking it down, already providing a sample repo! :green_heart: This is truly great! :rocket:

The bugs will never stop coming with this release will they :cry:

Anyhow. I also don't know how to best fix this. Our command guesser seems to first check for RSpec and then Minitest probably for that very reason.

We might also on initial start check the command name that is being run (although that gets muddy/murky with parallel tests I believe). So... I'm saying this seems like a hard problem :/ Querying Minitest for whether it's really running is probably the preferred solution (if possible) as I don't believe we want to check every other test framework for whether it's in used so maybe not minitest.

Other implementation notes, damn myself for not already introducing the full rails projects. We have a test_projects folder where you should be able to almost drop in your test project and then we can use it to write a cuke against it fixing the behavior. And make sure this never happens again :disappointed:

You PRing it would be mightily welcome, this is big enough of a bug though that I might try to clear some time in the coming days so a fix can be published ASAP. If you get there first, of course much welcome. I'll put a comment in this issue when I start working on it.

also cc: @adam12 as he seemed to know lots of things about minitest so might be of help :)

Thanks again for the great & nice report, it's greatly appreciated! :green_heart:

Ghost approves:

[adam12]

Instead of using the call to defined?, maybe we can setup a boolean accessor on SimpleCov that the Minitest plugin sets to true and that value is used as the guard instead.

[adam12]

I'll work on a PR for this.

[adam12]

@odlp @nathanvda Would you mind trying my branch which uses a different guard method?

https://github.com/adam12/simplecov/tree/alternative-at-exit-guard

I might be wrong and we might need to add the RSpec hook similar to what was mentioned above, but this seemed inline with the changes we've already made.

[odlp]

@adam12 I can confirm that branch restores the expected behaviour in the Rails+RSpec codebase I've been testing with 👍

[adam12]

Thanks @odlp

@PragTob I'll cleanup and submit as PR. Were you thinking about adding a Rails project for a test case?

[PragTob]

@adam12 yes, actually multiple. One with rspec, one with minitest etc... to make sure it all works in what's probably our must common use case. If you PR the changes I'm super happy adding the test projects myself :ok_hand:

[PragTob]

@adam12 if you're interested though, I imagine it like https://github.com/colszowka/simplecov/tree/master/test_projects/parallel_tests and the accompanying https://github.com/colszowka/simplecov/blob/master/features/parallel_tests.feature (minus the wips due to the other issue :sweat_smile: )

Long term vision is to have these for lots of common setups. Like Rails 5.2, Rails 6, some amount of different test frameworks. Rails + parallel tests.

All the fun to avoid these regressions in the future and also to have a framework to more easily reproduce bugs and write features/specs against it.

[PragTob]

Also: @adam12 thank you :green_heart:

[PragTob]

Fixed by #875 / #874 - releasing now. Thank you a lot everyone!