Rspec-core: Include line numbers as well as index in output

[xaviershay]

When RSpec outputs index format (rspec ./test_spec[1:1]) it is not parseable by humans or non-rspec computers. This affects things like vim plugins that automatically jump you to the failing test.

Needed:

• Repro. I've had reports of this on our large app but when I do a minimal spec it outputs line numbers in the failure. Under what circumstances does RSpec output the index format?
• Specific examples for vim plugins and the like that use this information.

Suggestions:

1. Include line number AND index: test_spec:2[1:1]. This is kind of a mess but would cover needed use cases.
2. Provide an interface (CLI) to "convert" index format to line number.

[myronmarston]

The logic that determines what to print for the example is here:

      def rerun_argument_for(example)
        location = example.location_rerun_argument
        return location unless duplicate_rerun_locations.include?(location)
        conditionally_quote(example.id)
      end

The idea is that RSpec prefers printing line numbers but they don't always uniquely identify an example, so it falls back to the example id if the line number is not sufficiently unique. This non-uniqueness primarily happens in two cases. One is when the user dynamically defines multiple examples in a loop:

RSpec.describe User do
  [:email, :password].each do |attr|
    it "requires a value for #{attr}" do
      user = build_user(attr => nil)
      expect(user).not_to be_valid
    end
  end
end

In this example, spec/models/user_spec.rb:3 identifies 2 different examples.

The other case is when the user uses a shared example group:

RSpec.shared_examples_for "requires a value for" do |attr|
  it "requires a value for #{attr}" do
    record = build_record(attr => nil)
    expect(record).not_to be_valid
  end
end

RSpec.describe User do
  alias build_record build_user
  it_behaves_like "requires a value for", :email
  it_behaves_like "requires a value for", :password
end

Again, the location (spec/models/user_spec.rb:2) identifies two different examples.

When I added example ids to RSpec I remember thinking about this, and being concerned that printing the example id instead of the location in the re-run list could be a step back for some folks that use it to go the spec in their text editor. However, given the list's primary purpose of providing commands for the user to copy and paste in order to re-run a specific example, I thought it was important that what is printed always uniquely identifies the example. At the time I considered adding a config option that would allow the user to pick whether locations or ids were printed but decided on the current compromise instead.

Include line number AND index: test_spec:2[1:1]. This is kind of a mess but would cover needed use cases.

This is an interesting idea I hadn't considered. RSpec would probably need to ignore the line number part, though, and that feels weird to have a throw-away part of what's printed that RSpec ignores.

Actually, there's a bigger problem with this idea: the _location_ of the spec might not be the file that's a part of the id that RSpec prints. Specifically, a spec can be defined in a shared example group in one file (e.g. in spec/support) and included in another file. In this kind of situation, the spec location might be spec/support/shared_examples.rb:2 whereas the example id (used for reruns) would be spec/models/user_spec.rb[1:1]. In fact, the location_rerun_argument method used above takes this into account -- it uses the list of loaded spec files and uses location metadata on a parent if necessary.

Provide an interface (CLI) to "convert" index format to line number.

Interesting idea, but would anyone use it?

[nerdrew]

What I want: a line number that is close so I can start debugging. I can't read the [1:1] notation, and my vim error format config can't either ;)

[JonRowe]

What I want: a line number that is close so I can start debugging.

I kind feel we should restore this, as the previous behaviour would have been this...

I can't read the [1:1] notation, and my vim error format config can't either ;)

Well it could ;)

[myronmarston]

I kind feel we should restore this, as the previous behaviour would have been this...

The previous behavior had lots of problems for when the user tried to copy and paste it as a re-run command. One idea is that we could switch to a form like:

rspec path/to/spec.rb[1:1] # path/to/spec.rb:15 -- spec description

Then it would have a working rerun line as it does now (although, if we do this, I think we'd make it _always print the id for consistency) and also the spec location and description.

It might make the lines super long, though...

[JonRowe]

@myronmarston Wouldn't doing rspec path/to/spec.rb:15[1:1] be ok? It's kinda reduced and we could just use the example but other things could use the imprecise line number variant?

[myronmarston]

@myronmarston Wouldn't doing rspec path/to/spec.rb:15[1:1] be ok? It's kinda reduced and we could just use the example but other things could use the imprecise line number variant?

It creates a string that is trying to refer to an example in two different ways, and they aren't necessarily going to be in sync. I think dealing with it will have odd semantics.

A larger problem is one I was trying to explain above: the file path for the id might be entirely different than the file path for the location. For example, an example could be defined in a shared group at spec/support/shared_groups.rb:10. If the shared group is included in a group defined in spec/my_spec.rb, the example id will be something like spec/my_spec.rb[1:1] while the location will be spec/support/shared_groups.rb:10.

[envygeeks]

I was prompted to come here and give my support for getting the line number. Even inside of a loop or vi shared spec, we can trace it faster via the line in the spec than we can with the current output. We too have a shared example with a loop inside of it that leads to this complicated 1:1:1 stuff, if it failed at the behaves_like :shared_spec we could trace it dozens of times faster than with the current output because well, in order to parse the current output we have to parse the blocks and all that which can be hard, especially for this one spec that is 1000 lines because it's double cautious because it handles filesystem stuff.

[myronmarston]

@envygeeks: I very much want to provide line numbers for people who want it but I also know that other people use that primarily as a "rerun line" and want something printed that is guaranteed to run only the one example. We used to always print the _definition_ location of the spec but that (rightfully) caused users to report issues like #793 (see also: guard/guard-rspec#243 and rspec/rspec-rails#991). If we switched back to printing the _definition_ location, it would degrade the experience for other users.

So it seems like our options are:

1. Keep it how it is, causing a worse experience for people who want to use the summary to go to the definition location in their text editor
2. Revert to how it was before, causing a worse experience for people who want to use the summary to rerun exactly one failing example.
3. Find a way to print both in the summary line. Problem is, the file paths may not be the same so we'd have to either use an inconsistent format (e.g. printing two file paths sometimes and only one other times) or always print the full id and full definition location as separate things on the same line, leading to extremely long output. I suspect no one would be very happy with this solution.
4. Make it configurable. Given the summary has different uses and people have different needs and we can't please everyone with any default, this seems like the best approach to me.

Thoughts?

[envygeeks]

@myronmarston I like the configuration option tbh. I know a lot of people like convention over configuration but I like configuration with convention by default. That gives us the option to kick on line numbers on places like Travis or when we run RSpec manually and convention in the configuration defaults it to the current way any place we don't reconfigure it.

[ahills]

However, given the list's primary purpose of providing commands for the user to copy and paste in order to re-run a specific example

I've actually never once used it for that purpose. What's the quickest way to determine which example failed from the example ID notation?

[JonRowe]

Thats kind of hard to ascertain outside of RSpec as it refers to the example by how it was defined,
e.g.

RSpec.describe {}
RSpec.describe do
  context do
    it "spec"
  end
end

Would produce [2:1:1] for 2nd example group in file, 1st child example group, 1st example.

[xaviershay]

If the shared group is included in a group defined in spec/my_spec.rb, the example id will be something like spec/my_spec.rb[1:1] while the location will be spec/support/shared_groups.rb:10.

In theory though, we should be able to know the line number at my_spec.rb[1:1] ... pretty sure we have this in metadata already? full_path or full_description?

[JonRowe]

In theory though, we should be able to know the line number at my_spec.rb[1:1] ... pretty sure we have this in metadata already? full_path or full_description?

We used to do dump the line number, but it's problematic, as there are multiple ways of defining examples such that line number means nothing (e.g. putting examples in ruby loops, using shared examples etc).

[myronmarston]

In theory though, we should be able to know the line number at my_spec.rb[1:1] ... pretty sure we have this in metadata already? full_path or full_description?

The line number of what? The it_behaves_like call? That's not going to uniquely identify an example unless the shared group has only 1 example.

[xaviershay]

yeah, at least then you know which line is causing the whole thing to run. See earlier comment about wanting to just end up in basically the right area to debug. We already have a way to uniquely identify, here we're just talking about a usability hack. To Jon's comment, even if it's in a loop I at least want to get to the loop.

[JonRowe]

here we're just talking about a usability hack.

Yeah thats why I suggested mixing the two, we need the id in order to re run definitely so we can't just revert the id, but personally I don't see the harm in also providing a line number.

[naw]

@myronmarston wrote:

Find a way to print both in the summary line. Problem is, the file paths may not be the same so we'd have to either use an inconsistent format (e.g. printing two file paths sometimes and only one other times) or always print the full id and full definition location as separate things on the same line, leading to extremely long output. I suspect no one would be very happy with this solution.

I think having both would be great. Personally I don't think long output is a problem, but perhaps others disagree.

I frequently use the current behavior to re-run a single spec, but it definitely would be nice to have a line number (even if its for a different filename) to find the actual code, since it _is_ pretty hard to track that down yourself if you have a complex shared example setup.

Both are useful (even at the same time), so printing both is more useful than a configuration to switch between the two, in my opinion.

[myronmarston]

Both are useful (even at the same time), so printing both is more useful than a configuration to switch between the two, in my opinion.

If we added a config option for this, it would allow people who want both to configure it. So there wouldn't be any switching back and forth; just setting it to your preferred format, and leaving it. Something like this:

RSpec.configure do |c|
  c.rerun_failed_example_format = "%{id} %{location}"
end

[juanibiapina]

Personally I would just like to take one line of output from spec/examples.txt, run through a CLI and get the relevant file and line number.

Two examples:

rspec --parse --relevant-line-on-this-file "./spec/models/user_spec.rb[1:1:1]" 

which outputs: spec/models/user_spec.rb:20

or

rspec --parse --assertion-line "./spec/models/user_spec.rb[1:1:1]"

which outputs: spec/support/shared_examples.rb:10

That way I can tell my tools to just call this CLI and jump to the relevant location. If I need to run rspec, I can just take the original line and feed to rspec itself.

I believe this could be simple enough to satisfy both sides.

@myronmarston ?

[0xdevalias]

Is there any update on how to resolve this? I'd find @juanibiapina 's solution super useful. Just _something_ (it really doesn't matter what) that lets my human brain unwind that reference and end up somewhere (other than my head through my desk) that I can start debugging from.

[JonRowe]

No, no one has had any time to spare to implementing any of the ideas, you should be able to unwind the reference though, its a simple count through examples and example groups, and using:

rspec "./spec/path_to/your_spec.rb[1:1:1]" --format documentation

Will print the doc string of your example which should enable you to find it easily.

[hakunin]

I am naming my include_examples invocations to work around this issue a bit:

      shared_examples 'match results' do |message|
        it message do
          # ...
        end
      end

      context 'when XYZ' do
        let(:results) { ... }
        include_examples 'match results', 'returns last enabled dates'
      end