Truffleruby: Molinillo test suite order of magnitude slower than MRI

[segiddins]

Steps to reproduce:

git clone --recursive https://github.com/CocoaPods/Molinillo.git
cd Molinillo
rake bootstrap
bundle exec rspec

Running the specs takes ~5 seconds on MRI 2.5.1 on my machine, whereas with truffleruby it took 4.5 minutes. (But they all pass 🎆)

[chrisseaton]

Thanks very much for the report. Passing is great. Cold code (which is often specs) can be a bit slower than MRI on startup, but we will investigate what is going on here.

[dentarg]

Same thing for Addressable:

• truffleruby-19.3.1: ~80 seconds
• ruby 2.6.5: ~2 seconds

(Then there's the bundler issue adding another 4-5 minutes where ruby does it in ~4 seconds 😄)

[eregon]

@dentarg Thanks for the report.
It might be worth trying with https://github.com/ruby/setup-ruby + truffleruby-head to see if that helps. I think it could help quite a bit for the Bundler time.

The 4 seconds for bundle install on MRI is not really comparable, because that used a Bundler cache for most gems.

For the specs:

Finished in 43.68 seconds (files took 15.92 seconds to load)

That's a very long time to load, there is probably something going wrong there, we should investigate.
And even then that doesn't amount to 80s reported by TravisCI, maybe the code is loaded once outside of specs too?

[dentarg]

It might be worth trying with https://github.com/ruby/setup-ruby + truffleruby-head to see if that helps. I think it could help quite a bit for the Bundler time.

Yes, around 1m instead of 5m: https://github.com/dentarg/addressable/runs/434731081?check_suite_focus=true

[dentarg]

truffleruby-head

Finished in 33.31 seconds (files took 8.49 seconds to load)

ruby 2.6.5

Finished in 0.97906 seconds (files took 0.2932 seconds to load)

[eregon]

@dentarg Thank you for trying with GitHub Actions and truffleruby-head, these numbers look significantly better.

We should still take a look at loading time and why those tests are much slower on TruffleRuby.

[jaimerave]

I just found this issue while trying to add support for Truffleruby in Rubocop (https://github.com/rubocop-hq/rubocop/pull/8602). bundle exec rspec takes 1 minute 36.75 seconds under MRI 2.6.6 to run on my local machine while with Truffleruby it takes 61 minutes.

[jaimerave]

Rubocop tests suit running with the Github action for truffleruby-head takes one and a half hour to finish https://github.com/rubocop-hq/rubocop/pull/8602/checks?check_run_id=1037142103

[bjfish]

@jaimerave I've opened a new issue for this: https://github.com/oracle/truffleruby/issues/2081

[gogainda]

@segiddins on my machine now it runs in 40 sec, could you retest it, please?
There were some improvements done on performance side, so looks like it may not be an issue anymore

Finished in 38.49 seconds (files took 2.22 seconds to load)

[gogainda]

Latest update:

Finished in 10.99 seconds (files took 1.63 seconds to load)

[eregon]

@gogainda Thanks for the update, that looks pretty close to CRuby, so I think we can close this (most likely MultiTier by default helps).