Rubocop: Figure out why our tests run so slow on Ruby 2.4.0

[Drenmi]

I noticed this when switching over to 2.4.0 and trying to work on RuboCop. Boot up time of the test suite is extremely long, and the entire test run takes almost twice as long as on 2.3.3. At first I thought it was something with my local installation, but the problem persists on Travis as well. 😅

I checked with rspec-core, but it seems like the issue is with our test suite in particular.

For now I'm using 2.3.3 to develop, to avoid going insane. 😅

---

Expected behavior

Tests run in approximately the same time as on Ruby 2.3.3.

Actual behavior

Tests run in twice the amount of time.

Steps to reproduce the problem

Switch to 2.4.0 (that's rvm use 2.4.0 or rbenv global 2.4.0) and run the tests.

RuboCop version

$ rubocop -V
0.46.0 (using Parser 2.3.3.1, running on ruby 2.4.0 x86_64-darwin15)

[pocke]

Ruby 2.4

https://travis-ci.org/bbatsov/rubocop/jobs/189770391

rake command took 620.98 seconds.

Ruby 2.3.3

https://travis-ci.org/bbatsov/rubocop/jobs/189770390

rake command took 346.03 seconds.

In ruby 2.4, rake is slow... 🤔

[pocke]

I created docker images to test the problem in local machine.

FROM ruby:2.3.3

RUN git clone --depth 1 https://github.com/bbatsov/rubocop \
  && cd rubocop \
  && bundle install \
  && bundle exec rake install \
  && rubocop -V

FROM ruby:2.4.0

RUN git clone --depth 1 https://github.com/bbatsov/rubocop \
  && cd rubocop \
  && bundle install \
  && bundle exec rake install \
  && rubocop -V

# each the dockerfiles
$ docker build -t rubocop:2.4.0 .
$ docker build -t rubocop:2.3.3 .

First time, I executed rubocop command in rubocop's repository with two MRI version.

$ pwd # => /home/pocke/ghq/github.com/bbatsov/rubocop

$ time docker run -v $(pwd):/tmp/work -it rubocop:2.4.0 bash -c "cd /tmp/work && rubocop"
54.39s

$ time docker run -v $(pwd):/tmp/work -it rubocop:2.3.3 bash -c "cd /tmp/work && rubocop"
55.45s

From this results, I think rubocop command doesn't have performance problem.

Second time, I executed rspec command in rubocop's repository with two MRI version.

$ pwd # => /home/pocke/ghq/github.com/bbatsov/rubocop


$ time docker run -v $(pwd):/tmp/work -it rubocop:2.4.0 bash -c "cd /tmp/work && rspec"
628.71s

$ time docker run -v $(pwd):/tmp/work -it rubocop:2.3.3 bash -c "cd /tmp/work && rspec"
31.61s

RSpec has a difference of 20 times!

[Drenmi]

From this results, I think rubocop command doesn't have performance problem.

Yeah. RuboCop runs 1-2 seconds faster on my machine as well.

[bbatsov]

That's quite fascinating. I guess the problem should be in RSpec after all.

[Drenmi]

I guess if it gets fixed only in Ruby 2.4.1 only, we'll always have a slow running 2.4.0 build in the matrix on Travis. 😓

We should at least move 2.4.0 up towards the top in the configuration so that it starts early, much like how we treat JRuby. WDYT?

[deivid-rodriguez]

I had a similar problem a while ago and fixed it by manually patching the problematic ruby. In this case, the problem was a SEGV, not sure if a slow build is worth doing that but I thought I would mention it. This is what I did: https://github.com/deivid-rodriguez/byebug/commit/a2e0c9bae6ec3e92e843aac2cad22a055edc235f

[deivid-rodriguez]

Actually, patching ruby probably takes longer than the slow build, so... forget about it! :smiley:

[backus]

Ruby 2.4.1 is out and it will probably close this issue FYI.