Rspec-core: Allow order specs by modification time

[MatheusRich]

Subject of the issue

Quite often is useful to run tests that you've just modified. RSpec lets you to define custom ordering, so I have this snipped configured in my projects:

config.register_ordering(:global) do |items|
  items.sort_by { |item| -File.mtime(item.metadata[:absolute_file_path]).to_i }
end

I thought it might be useful to upstream this for more users to use this with an option --order modification_time. Thoughts?

Your environment

ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux-musl]

RSpec 3.9

• rspec-core 3.9.2
• rspec-expectations 3.9.2
• rspec-mocks 3.9.1
• rspec-rails 4.0.1
• rspec-support 3.9.3

[pirj]

This is interesting. However this neglects the previous ordering, be it defined or random that relates to the order of the examples inside the spec, right?
I've seen suites really depending on defined order, wouldn't this break it? Do you think it would be possible to add this sorting, which is really to order the spec files, a second-tier ordering?

Another idea would be only be running the recently modified specs, as reported by, say, git.

Doesn't guard achieve the same goal for you by the way?

[JonRowe]

This is an interesting feature suggestion, although its so easy to implement I'm sort of on the fence as saying it could easily be a plugin rather than a core feature.

However this neglects the previous ordering, be it defined or random that relates to the order of the examples inside the spec, right?

Yes thats the point.

I've seen suites really depending on defined order, wouldn't this break it?

Yes but then they wouldn't choose this ordering scheme, its an alternative not a combination (also those suites should be fixed).

Doesn't guard achieve the same goal for you by the way?

I see guard as orthogonal to this, it reruns files as they change, not running the whole suite by the modification times. This would allow you (combined with --next-failure) to find the most recent failing spec, which you might not have just changed, but could have changed say, yesterday.

Where it potentially falls down is it has no knowledge of library file changes, so it'll always be local filesystem spec mtime, and not true age, or wether its actually been covered.

[benoittgt]

I love this suggestion and never thought about it. I am in favor of having this in RSpec itself. 😄

[MatheusRich]

My goal was really to combine this with --next-failure. In my experience, modified specs are the most susceptible to failures.

[pirj]

One thing you may be interested in is https://github.com/toptal/crystalball/
It's a sophisticated tool that infers the specs that might break depending on your code changes. One of its most naïve strategies is to run the modified specs.

[MatheusRich]

So, should I open a PR for this? I'm not sure if the whole core-team agrees with the feature :smile:

[pirj]

Can you implement it as a plugin, rather than baking in into RSpec? Will you have to monkey-patch internals, or does RSpec provide you necessary interface to make such an extension, @MatheusRich ?

[MatheusRich]

To be honest, I don't know. Probably there's a plugin support. But I'm thinking if there's any value in making it a separated gem. I mean, I'm lazy, so I'd prefer to setup guard or crystalball than just add a gem for this simple thing.

[pirj]

Do you think there is something actionable here @MatheusRich or this ticket can be closed?

[MatheusRich]

I think the maintainers aprooved the feature, so I'm keen to implementing it

[JonRowe]

I don't feel strongly against it, so if @benoittgt wants to champion it I'm fine with that @pirj