Rspec-core: Backtrace exclusion doesn't work in RSpec 3.5.4

[marko-avlijas]

I have this part in my spec_helper.rb, starting with default configuration

# This option will default to `:apply_to_host_groups` in RSpec 4 (and will
# have no way to turn it off -- the option exists only for backwards
# compatibility in RSpec 3). It causes shared context metadata to be
# inherited by the metadata hash of host groups and examples, rather than
# triggering implicit auto-inclusion in groups with matching metadata.
config.shared_context_metadata_behavior = :apply_to_host_groups

puts "HERE!!!"

# removes too long exception backtrace messages
config.backtrace_exclusion_patterns = [
  /rspec/,
  /\/lib\d*\/ruby\//,
  /org\/jruby\//,
  /bin\//,
  /gems/,
  /spec\/spec_helper\.rb/,
  /lib\/rspec\/(core|expectations|matchers|mocks)/
]

All of these are ignored. I have even put rspec out of desperation. Backtrace is so long that I have to use mouse and scroll up every time. :(

Here is the output I get

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
$ rspec -v
3.5.4
$ rspec
HERE!!!
/home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:724:in `method_missing': `get_property` is not available on an example group (e.g. a `describe` or `context` block). It is only available from within individual examples (e.g. `it` blocks) or from constructs that run in the scope of an example (e.g. `before`, `let`, etc). (RSpec::Core::ExampleGroup::WrongScopeError)
    from /data/sites/scripts/sw_scripts/ruby/gallery/spec/specs/specs_spec.rb:33:in `block (3 levels) in <top (required)>'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/shared_example_group.rb:36:in `class_exec'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/shared_example_group.rb:36:in `block in include_in'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:797:in `with_frame'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/shared_example_group.rb:35:in `include_in'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:373:in `find_and_eval_shared'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:341:in `include_examples'
    from /data/sites/scripts/sw_scripts/ruby/gallery/spec/specs/specs_spec.rb:68:in `block (3 levels) in <top (required)>'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `module_exec'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `subclass'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:258:in `block in define_example_group_method'
    from /data/sites/scripts/sw_scripts/ruby/gallery/spec/specs/specs_spec.rb:66:in `block (2 levels) in <top (required)>'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `module_exec'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `subclass'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:258:in `block in define_example_group_method'
    from /data/sites/scripts/sw_scripts/ruby/gallery/spec/specs/specs_spec.rb:14:in `block in <top (required)>'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `module_exec'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `subclass'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:258:in `block in define_example_group_method'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/dsl.rb:43:in `block in expose_example_group_alias'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/dsl.rb:84:in `block (2 levels) in expose_example_group_alias_globally'
    from /data/sites/scripts/sw_scripts/ruby/gallery/spec/specs/specs_spec.rb:6:in `<top (required)>'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `load'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `each'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
    from /home/marko/.gem/ruby/2.3.1/bin/rspec:22:in `load'
    from /home/marko/.gem/ruby/2.3.1/bin/rspec:22:in `<main>'

[marko-avlijas]

Is there a quick and dirty way to run with no backtrace at all? Would be more helpful to just read the message instantly even if I lost the line number which caused the error.

In future I think that better settings than bunch of complex regexes is just to limit backtrace length to 5 lines or so. I just need the line number that caused the error and it's usually in first 2 lines.

[myronmarston]

I believe you've configured RSpec to show the full backtrace. This can be done in a few ways:

• --backtrace or -b in .rspec
• --backtrace or -b in ~/.rspec
• --backtrace or -b in .rspec-local
• config.backtrace = true in a Ruby file loaded as part of your suite

This option provides a way to easily see the full backtrace for a specific run (which is why it overrides the exclusion patterns you have set--no matter what exclusion you've set, there are occasions where you really need to see the entire backtrace). In general, I recommend you only pass it via the command line, and do not commit it to a file as it sounds like must have happened for you.

[marko-avlijas]

From project directory.

$ cat .rspec 
--color
--require spec_helper
$ cat .rspec-local
cat: .rspec-local: No such file or directory
$ cat .~/rspec
cat: .~/rspec: No such file or directory
$ grep -nir backtrace .
./spec/spec_helper.rb:20:  config.backtrace_exclusion_patterns = [
$ 

I see a typo here so I have repeated it

$ cat ~/.rspec
cat: /home/marko/.rspec: No such file or directory

And this:

$ rspec --no-backtrace
invalid option: --no-backtrace

Please use --help for a listing of valid options

[myronmarston]

After re-reading your original issue, I see what the problem is now: at spec/specs/specs_spec.rb:33, you have get_property directly in an example group block (e.g. a describe or context) but this method is only available for individual examples and needs to be put in an example block (e.g. an it block). The consequence of this is that this error is _not_ happening while your specs are running--it's happening when your spec file is loaded, because the example group blocks are evaluated eagerly when the files are loaded.

In RSpec 3.5 and before, exceptions that occurred while loading spec files were not handled in anyway whatsoever--and ruby prints the entire backtrace (it has no reason to use RSpec's backtrace filtering...). In RSpec 3.6 (of which 3.6.0.beta1 has been out for a bit), we now handle this much better: we rescue errors that occur while spec files are being loaded and filter the backtrace using the filtering options.

So, in summary:

• The fact that you are seeing this for a load-time error does not indicate backtrace filtering is not working for example failures (the only place where it has ever worked for RSpec 3.5 and before).
• If you want it to work for this situation, upgrade to RSpec 3.6.0.beta1.
• Or, just fix this specific error. The backtrace filtering for failures in your examples should still work.

[marko-avlijas]

How do I use code in example groups? I don't want it to be in module Helper becaues it's local to that spec and it helps me clean up examples.

describe Specs::Specs do
  shared_context 'common' do
    def get_property(type)
      subject.send(type)
    end
  end

  describe 'attributes not needing conversion' do
    include_context 'common'
    get_property(:weight)
  end
end

How to make it so I can actually use get_property from within nested example groups?

[myronmarston]

get_property is available within nested example groups -- but it must be called from within _examples_ in nested groups, and not within the body of the group itself. This works, for example:

RSpec.describe 'Specs::Specs' do
  shared_context 'common' do
    def get_property(type)
      subject.send(type)
    end
  end

  describe 'attributes not needing conversion' do
    subject { [1, 2, 3] }
    include_context 'common'

    it 'works in an example in this example group' do
      expect(get_property(:length)).to eq 3
    end

    context "in a nested group" do
      subject { Time.now }

      it 'works in an example in a nested group, too' do
        expect(get_property(:year)).to eq 2016
      end
    end
  end
end

Notice I am calling get_property from within examples in the group and the nested group, not from the body of the nested group itself. If you want to define a method that can be called from the body of a group, you can do that, too, but you have to define the method as a singleton method (or "class method"):

RSpec.describe 'Specs::Specs' do
  shared_context 'common' do
    def self.define_lets(lets)
      lets.each { |name, value| let(name) { value } }
    end
  end

  describe 'attributes not needing conversion' do
    include_context 'common'
    define_lets foo: 12, bar: "a"

    it 'works in an example in this example group' do
      expect(foo).to eq 12
      expect(bar).to eq "a"
    end

    context "in a nested group" do
      define_lets bazz: 19

      it 'works in an example in a nested group, too' do
        expect(bazz).to eq 19
      end
    end
  end
end

In general, example groups and examples have the same relationship as a class and an instance of the class. The body of the example group (the describe block) defines a class, and individual examples are run within an instance of the class. This is why methods defined via def method_name are available for use within examples but not within the group body (just like how methods defined via def method_name in a class body work) while methods defined via def self.method_name are available for use within the example group body but not the examples themselves (again, this is just like how class methods work in a class body).

For your specific example--get_property--it does not make sense to call it from within the group body. There are two subject methods:

• In the group body, it _defines_ the subject, and does not return a meaningful value. (It exists only for the side effect of defining the subject)
• In an example it returns the subject

Since subject in a group exists only for the side effect and has no meaningful return value, subject.send(type) in that context is going to fail, and I'm not even sure what you would be trying to do with it in a group body. But within an individual example it works just fine.

[marko-avlijas]

I think it makes perfect sense. I don't really need subject, just described_class.new

What I have is (I need to refactor that first example to be generec)

shared_examples_for 'single unit type' do |type, default_unit|
  it "allows setting #{type} in '#{default_unit}'" do
    subject.production.set(1_000, 'units')
    expect(subject.production.value).to eq(1_000)
    expect(subject.production.unit).to eq('units')
  end
  ...
end

describe ':production' do
  include_examples 'single unit type', :production, 'units'
end

describe ':engine_volume' do
  include_examples 'single unit type', :engine_volume, 'liter'
end

I am going to have a bunch of these and what I'd prefer is to just pass along type and to ruby read default unit for that type automatically instead of manually specifying 'liter' and 'units'.
All I need to do is access it is this code from example groups

def default_unit_for(type)
  # 
end    

This seems impossible without moving it into a module, but I need default_unit_for only in 1 spec file.

[myronmarston]

I don't understand what you are trying to accomplish.

[marko-avlijas]

shared_examples_for 'single unit type' do |type, default_unit|

Here I want to remove parameter default_unit and instead get it automatically.

[myronmarston]

Where would you like it to get default_unit from? I don't understand that from what you've provided so I can't advise you.

Also, why is making it more implicit desirable? Passing an argument is very explicit and easy to understand; having something default based on an implicit mechanism is harder to understand.

[marko-avlijas]

Code that would get me the unit would be this:

described_class.new.send(type).class.allowed_units[type]

I wanted to do it so I don't have to change the tests if I change unit in code, but maybe that's not the brightest idea.

[myronmarston]

This should work:

shared_examples_for 'single unit type' do |type|
  let(:default_unit) { described_class.new.send(type).class.allowed_units[type] }

  it "does something with default unit" do
    do_something_with(default_unit)
  end
end

[marko-avlijas]

Yes but it doesn't work in string description of example. What I have is:

it "allows setting #{type} in '#{default_unit}'" do
  ...
end

[myronmarston]

That's because let defines an instance method for use within examples, but isn't available to be called from the group itself. After all, a big part of what let does is memoize the value for each example, but that can't be done when you aren't in an example.

If you want to access it from group context, as you are there, then do this:

shared_examples_for 'single unit type' do |type|
  default_unit = described_class.new.send(type).class.allowed_units[type]

  it "allows setting #{type} in '#{default_unit}'" do
    do_something_with(default_unit)
  end
end

It's just ruby, so you can create local variables anywhere and access them from that scope and also any blocks in the scope.

Be aware, however, that we usually recommend against instantiating the class you are testing within the body of the example group. While it might work just fine in your case, the fact that it is instantiated at spec load time often has some significant downsides, and these may not be obvious:

• When you are filtering to just one example (e.g. via line number or using focus filtering or whatever), you probably don't want it instantiating other classes that are only involved in specs you aren't running. But since the group is evaluated eagerly (in order to define examples at load time) it'll happen regardless of whether or not any specs in that context are going to run. If instantiation is slow, it can make loading your spec files really slow even if you aren't running the examples.
• If the class interacts with any external state (e.g. the database) that is usually managed on a per-example basis (such as rolling back a transaction after every spec), then using it in the body has the potential to bypass your sandboxing mechanism. For example, if instantiating the class causes a record to be put in the DB, it'll happen outside the context of your per-example transactions which could cause all sorts of problems.

[marko-avlijas]

 default_unit = described_class.new.send(type).class.allowed_units[type]

I could do this but I already have a function default_unit_for that does this for examples in that spec file so I'd like to avoid duplication by somehow reusing that function, or creating self.default_unit_for but that results in some errors like:

/home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/formatters/base_text_formatter.rb:69:in `write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/formatters/base_text_formatter.rb:69:in `puts'
    from /home/marko/.gem/ruby/2.3.1/gems/rspec-core-3.5.4/lib/rspec/core/formatters/base_text_formatter.rb:69:in `close'

It would be nice to be able to reuse functions in examples and example groups.

Thank you very much for your help. I have to leave for a meeting now.

[myronmarston]

That error is very odd. Sounds like a bug in RSpec. Defining a helper method (either via def foo or def self.foo) should not cause a broken pipe error. Can you boil that down to a reproducible example an open an issue?

It would be nice to be able to reuse functions in examples and example groups.

The simplest way to do that is to put the method in a module and then you can both include and extend the module to make it available to both groups and examples:

module UnitHelpers
  def default_unit_for(type)
    # ...
  end
end

RSpec.describe MyClass do
  extend UnitHelpers
  include UnitHelpers

  # ...
end

[marko-avlijas]

Can you boil that down to a reproducible example an open an issue?

I don't have time today and code is deleted, but try creating default_unit_for(type) and self.default_unit_for(type) inside example group. I don't remember if I did it in same context or one was nested.

I remember this piece of code

default_unit_for(type)
  self.default_unit_for(type)
end

or this

default_unit_for(type)
  self.class.default_unit_for(type)
end

I have called both of these inside body of shared_examples_for and inside those examples and calling one or both of them crashed it.

It is possible that this code was there as well (subject in static class)

def self.default_unit_for(type)
   subject.send(type).class.allowed_units[type]
end

[myronmarston]

Please put together a github repo with code that triggers this bug. We need something we can clone and run to troubleshoot. We generally don't have the time to put together such a thing ourselves for something like this. In my experience, we generally can't reproduce if we try to put it together.

[marko-avlijas]

OK I'll try tomorrow. Code is deleted so I can try to fiddle with what I wrote in post above.

[marko-avlijas]

I couldn't reproduce the issue.

[myronmarston]

Ok. Closing.