Rspec-core: Feature request: allow running the same test multiple times with custom data

[bootstraponline]

For Selenium testing, it's necessary to run the same test multiple times with different data each time (desired capabilities control the execution environment). Currently to implement this requires instance eval. It'd be nice if rspec-core allowed running example groups multiple times without hacks. The existing dedupe logic means duplicate example groups are ignored.

[myronmarston]

The standard way to run a test multiple times with different data is to use dynamically define them in an each block:

RSpec.describe "Dynamic tests" do
  [value_1, value_2].each do |val|
    it "runs some test logic with #{val}" do
      run_test_logic_with(val)
    end
  end
end

Any reason that doesn't work for your use case?

[bootstraponline]

Yes, I have 100s of tests and the values are constantly changing. Wrapping all the tests in an each doesn't scale. The config looks like this:

SauceRSpec.config do |config|
  config.caps = [
    Platform.windows_8.firefox.v37,
    Platform.mac_10_11.safari.v8_1
  ]
end

Each RSpec test is executed in duplicate (once for safari, once for firefox).

In the Sauce Ruby gem an environment variable is set and each test is executed using a separate process which is also an unfortunate hack.

[JonRowe]

I think you'd be better off with a custom runner that changes the config and reruns the specs personally

[bootstraponline]

I'm already using a custom runner for parallel execution. I don't think I can use two RSpec runners. By creating the tests on World register then the change is independent of the runner.

[JonRowe]

You can't use two runners but you could further customise your existing one.

[bootstraponline]

I'd have to fork the parallel execution gem and modify every RSpec runner which is less than ideal.

[JonRowe]

No you'd just have to modify the one runner you use... This is a fairly custom thing you're requesting so I think you're going to have to write some custom code to do it.

As @myronmarston says the usual way to run tests multiple times is define the tests multiple times using an iterator, while it isn't ideal in your case to do so because you'd have boilerplate in every file you could actually set it up that way fairly easily. If you really wanted to ease this you could create a custom DSL (Sauce.describe?) for yourself that calls RSpec.describe once for every environment combination (which a different name ideally) and just use that in your specs.

Either way I'm closing this for now as I don't think it's something that belongs in rspec-core

[bootstraponline]

No you'd just have to modify the one runner you use...

The goal is to offer a solution that works with any of the parallel execution gems which all define their own runner. test-queue alone has at least 3 runners for rspec. Forking these and maintaining custom runners isn't a viable option.

Sauce.describe means people would have to modify their existing tests which isn't acceptable.

This is a fairly custom thing you're requesting so I think you're going to have to write some custom code to do it.

I think the custom code which patches rspec is the most maintainable fix for now. It's unfortunate that rspec-core doesn't have a more flexible API. A hook into the World register method would enable this use case in a more general way without monkey patching.

[JonRowe]

If you feel like adding that hook and it does it in such a way that doesn't
impact the performance for the general use case please feel free to open a
PR

[bootstraponline]

I think the following code would have minimal performance impact while making it easier for 3rd party code to manipulate example groups before they're registered.

If this approach seems fine then I'm happy to submit a PR with tests. If the use case is too unique though then I understand not wanting it in core.

# @api private
#
# Prepare an example group for registration.
# Hook used by 3rd party code.
def before_register(example_group)
  example_group
end

# @api private
#
# Register an example group.
def register(example_group)
  example_group = before_register(example_group)
  example_groups << example_group
  @example_group_counts_by_spec_file[example_group.metadata[:file_path]] += 1
  example_group
end

[myronmarston]

We're definitely open to adding new APIs to address your needs, as @JonRowe said, but I still don't understand why the normal "define multiple examples in a loop" approach doesn't work. Can you paste together a more realistic example of what you're trying to do?

[bootstraponline]

Sure. I'll work on a better example.

[JonRowe]

Almost, it'd be better if the before_register hook was a series of blocks that are registered in configuration (on a public api) but the default is empty and thus just a no op. Kind of like how we've done the verifying double callbacks for mocks...

https://github.com/rspec/rspec-mocks/blob/1826212a66598b171fece2998f68998e4df940e1/lib/rspec/mocks/configuration.rb#L109

[bootstraponline]

Thanks for clarifying the correct approach. I agree that's better and I'll update my code.

[bootstraponline]

I haven't forgotten about this. I expect to submit a PR within the next few weeks.

[bootstraponline]

@myronmarston Here's a full example of the setup. It's using the same gems / rspec setup as the test suite I manage for work. In sauce_helper I'm able to run all rspec tests on multiple platforms without having to change anything in the specs or rspec config. This is possible due to the World.register patch in sauce_rspec. The idea is for Sauce to enable their customers to use this approach so anything that involves modifying specs doesn't work.

It's taken me a bit longer to update the other parts of the sauce gems, for example the custom formatter. I still plan on submitting a PR for before_register.

[bootstraponline]

I've submitted https://github.com/rspec/rspec-core/issues/2094

[elrayle]

I tried to use the original suggestion by @myronmarston but it is not working as I had hoped. Here is a simplified test following that pattern...

RSpec.describe "Dynamic tests" do
  [1, 2].each do |val|
    let(:my_val) { val }

    it "runs some test logic with #{val} expecting 1" do
      puts "Test: runs some test logic with #{val} expecting 1"
      puts "  val=#{val}"
      puts "  my_val=#{my_val}"
      expect(my_val).to eq 1
    end
    it "runs some test logic with #{val} expecting 2" do
      puts "Test: runs some test logic with #{val} expecting 2"
      puts "  val=#{val}"
      puts "  my_val=#{my_val}"
      expect(my_val).to eq 2
    end
  end
end

This prints...

Test: runs some test logic with 2 expecting 1
  val=2
  my_val=2
F
Test: runs some test logic with 1 expecting 1
  val=1
  my_val=2
F
Test: runs some test logic with 2 expecting 2
  val=2
  my_val=2
.
Test: runs some test logic with 1 expecting 2
  val=1
  my_val=2
.

Note that my_val is NOT set based on the value of val. I expect my_val and val to always have the same value, but my_val is always 2.

[JonRowe]

@elrayle Thats because you can't use let in this fashion, its the equivalent of:

let(:my_val) { 1 }
let(:my_val) { 2 }

You are overwriting the previous definition. You either need to use val directly, or wrap each unit in a describe or a context.