Simplecov: Number of inspected lines decresed in 0.15.0

[Ana06]

When updating simplecov from 0.14.1 to 0.15.0 (https://github.com/openSUSE/open-build-service/pull/3595) the coverage in codecov (https://codecov.io/gh/openSUSE/open-build-service/pull/3595?src=pr&el=tree) gets significantly decreased. This is strange because lines like the end of the methods appear now uncovered. I also run simplecov locally and I saw that the number of code inspected gets decreased, so could be that the lines that codecov marks now at uncovered are actually not inspected.

Any idea what this happens?

[PragTob]

First thanks a lot for the issue report :tada:

Hm that's strange. Theoretically, the number of lines should only have changed for files that aren't loaded as we modified how these lines are reported. Could that be the case there?

Great that it's open source, so I might take a look and figure that out. Honestly, I might not have time too soon. If you could give me one specific file as an example to look at and see if there's a fix that'd be easier :)

Also :+1: for @depfu usage!

[Ana06]

@PragTob

Hm that's strange. Theoretically, the number of lines should only have changed for files that aren't loaded as we modified how these lines are reported. Could that be the case there?

I doubt it, because there are not complete file uncovered.

Great that it's open source, so I might take a look and figure that out. Honestly, I might not have time too soon. If you could give me one specific file as an example to look at and see if there's a fix that'd be easier :)

Sure, I would say this two files are representative:

• end and some other "ending" parts are now uncovered (this is the case in most of the files): https://codecov.io/gh/openSUSE/open-build-service/compare/7e8c140fd74a718efb01fb013497930bb16bf8f4...9d51081aae7a8fe8e8414b8d3f6440f3aca47c70/src/src/api/app/helpers/webui/webui_helper.rb

• strange case, because there are some new uncovered lines, but also new covered lines: https://codecov.io/gh/openSUSE/open-build-service/compare/7e8c140fd74a718efb01fb013497930bb16bf8f4...9d51081aae7a8fe8e8414b8d3f6440f3aca47c70/src/src/api/app/models/bs_request/data_table/params_parser.rb

[PragTob]

Had a quick look today and noticed that the simplecov setup is off, putting off further investigation until fixed so I don't spend time chasing shadows :)

[bgeuken]

Thanks a lot for looking into this. We fixed our simplecov setup (https://github.com/openSUSE/open-build-service/pull/3816) as you suggested. The coverage decreases is lower now, but still see a drop of -1.64% when upgrading to simplecov 0.15.1.

In most of the cases the coverage decreased, because the closing 'end' of a method or class description is not counted as covered anymore. Though the method content still is.

Here would be one example of this.

[PragTob]

Thank you very much!

That does seem very strange indeed... and definitely shouldn't happen :scream: Will take another look (right now if I can get it done quickly)

[PragTob]

:wave:

So at the point where it became clear I had to install mysql2 I gave up on setting the application up for now. Another thing I noticed is that it appears that you might start simplecov 2 times (both from rails_helper and test_helper) which might lead to some missing coverage as mentioned in the README. You could use the support/coverage you already have in place and also use it from test_helper.rb.

Alternatively the suggested way is to use .simplecov :)

Let me know if this helps, if not I guess I'll have to install mysql (the reported behaviour doesn't appear in our fairly thorough tests so replication outside the app doesn't seem trivial, if you could do it it'd be greatly appreciated but I can see how that's also a huge investment especially given that the original is already OSS).

Thanks!
Tobi

[hennevogel]

@PragTob I've created a minimal setup that shows this -> https://github.com/hennevogel/simplecovbug

[hennevogel]

[hennevogel]

Maybe somehow a combination of autoloading and instantiating with with subject?

[PragTob]

@hennevogel thanks a lot! I can't promise when I'll get to it but this definitely helps a ton! :)

[hennevogel]

@PragTob thank you 🤠

I really think something weird is going on. I can't pinpoint it (yet) but we have another issue with the same module that is involving a third directory level like

app/models/some_thing/another_thing.rb
app/models/some_thing/another_thing/third_thing.rb

where coverage of third_thing.rb is measured, coverage of another_thing.rb not at all...

[PragTob]

@hennevogel

Hi there!

Got to looking at it, and I'm a bit unsure: Where is the bug? :grin:

I.e. the end isn't marked as covered but as it's not a "signficant line of code" it is ignored by the standard library coverage tool - it gets the same signifcance as a comment:

tobi@speedy ~/github/simplecov $ cat try.rb 
# some comment
def method
  42
end
tobi@speedy ~/github/simplecov $ cat cover.rb 
require "coverage"
Coverage.start
require_relative "try"
p Coverage.result
tobi@speedy ~/github/simplecov $ ruby cover.rb 
{"/home/tobi/github/simplecov/try.rb"=>[nil, 1, 0, nil]} # array of the code lines and their hits in the time

So if we're talking about end not being covered that's expected to the best of my knowledge :)

edit: the other thing you mentioned above... that _could_ be some sort of directory mixup but I'd have to look at it

[PragTob]

Can't discover the bug, hence closing. If you disagree/have more hints where the bug is please let me know!