Simplecov: Incorrect results on Travis CI

[michaelmior]

Locally I get ~83% coverage, but I only get ~52% on Travis CI. Perhaps it's not a SimpleCov bug, but hoping someone might have some ideas. Using SimpleCov 0.9.1. Ruby version is identical in both environments (ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]). Tests are run with RSpec 3.1.0. Any suggestions appreciated :)

[michaelmior]

Strangely, I'd like to add that using the Code Climate gem seems to work directly despite the fact that it's using the same version of SimpleCov under the hood.

[bf4]

I can't really diagnose your problem without any code or more information, but it may be related to running parallel specs, the command you run the tests with locally vs. on ci, or your merge timeout, if any.

Please see https://github.com/colszowka/simplecov/issues/340 and report back.

[michaelmior]

Sorry for the lack of details. Hopefully the points below help

• This is not a Rails app
• My command locally is the same as on Travis CI bundle exec rspec --tag ~disabled
• I have no .simplecov file (however, I added one with my config from spec_helper.rb and this didn't change anything)
• My tests are not run in parallel

[bf4]

Without more code there's really nothing I can do.

[sunnyrjuneja]

@michaelmior I had a similar problem with Circle CI. I think the problem might be that the gems are being installed in your apps directory and they're being added to simple cov report. I was able to resolve this using an add_filter "/vendor/" but if your directory is different, you might have the same problem.

[michaelmior]

@whatasunnyday Thanks! That did the trick. I forgot that by default Travis configures bundler to install dependencies in vendor. Given how common the usage of Travis CI is these days, perhaps that's worth mentioning in the documentation somewhere.

[sunnyrjuneja]

@bf4 Would you be interested in a PR? I had the same problem today and spent 10 mins figuring it out.

[bf4]

Sure. I've had this problem in other situations as well when I forget that I've bundled gems in the current dir, like when I use bundle standalone ( add_filter 'bundle' ).

It looks like only the Rails profile excludes vendor/bundle by default I'm not sure why that would be.

@colszowka @sferik @xaviershay Any reason not to put these in the test_frameworks config? The commit that added it only to the Rails profile doesn't make me think otherwise.

Per travis docs

If you have custom Bundler arguments, and these include the --path option, Travis CI will use that path. If --path is missing but --deployment is present, it will use vendor/bundle.

Otherwise it will automatically add the --path option. In this case it will either use the value of the environment variable BUNDLE_PATH or, if it is missing, vendor/bundle.

(via the bash-fu --path=${BUNDLE_PATH:-vendor/bundle} that falls back to vendor/bundle if BUNDLE_PATH isn't set)

[xaviershay]

Sorry I missed your question when you first asked it. I was just bitten by this today. I'd support adding add_filter '/vendor/bundle/' to the appropriate default config.

[xaviershay]

For completeness:

• Failing build: https://travis-ci.org/square/ruby-tree_filter/jobs/54964333
• Fix: https://github.com/square/ruby-tree_filter/pull/1
• Was able to repro locally by bundle install --deployment

Going to re-open this ticket.

[sferik]

I'd support adding add_filter '/vendor/bundle/' to the appropriate default config.

This has been added and released in version 0.10.0. Please reopen this issue if that doesn’t solve it for you.