Rspec-core: expect { block }.to output('something').to_stderr swallows specs after it

[marko-avlijas]

This is reduced test case:

require 'author/authors_helper'

describe AuthorsHelper do
  describe '.load_csv' do
    let(:authors) { AuthorsHelper.authors }

    def set_csv_path(filename)
      @csv_path = File.expand_path("fixtures/#{filename}",
        File.dirname(__FILE__))
      allow(AuthorsHelper).to receive(:csv_path).and_return(@csv_path)
      # clear authors from last test
      AuthorsHelper.authors.clear
    end

    def load_csv
        AuthorsHelper.load_csv
    end

    context 'when csv is valid' do
      before (:each) do
        set_csv_path('valid.csv')
        load_csv
      end

      it 'loads all authors' do
        expect(authors).to include(have_attributes(name: '2M-Designs'))
        expect(authors).to include(have_attributes(name: '3dCarbon'))
        expect(authors).to include(have_attributes(name: 'Designer Name'))
      end
    end

    context 'when csv is corrupt outputs to stderr when' do
      it 'author name is repeated twice'
      it 'both require feild is missing and author name is repeated twice'

      it 'filename is missing' do
        set_csv_path('missing_filename.csv')
        expect { load_csv }.to output('Muahahah').to_stderr
      end
    end
  end
end

2 Issues here:
1) Last spec 'filename is missing' should not pass because no code outputs 'Muahahah'

2) Last spec swallows all tests after it.
So if I run like this I get 4 examples, 0 failures, 2 pending, but if I put pending specs after 'filename is missing' rspec will show 2 examples, 0 failures

What's wrong here and how to fix it?

[marko-avlijas]

Alright this was an error because I had abort in code I was testing.
I have made a capture_stderr method to deal with this.

def capture_stderr(&block)
  original_stderr = $stderr.dup
  output_catcher = StringIO.new
  $stderr = output_catcher
  begin
    yield
  rescue SystemExit
  ensure
    $stderr = original_stderr
  end
  output_catcher.string
end

I am using it like this:

error = capture_stderr('missing_filename.csv')
expect(error).to include('part of error message')

[JonRowe]

Yeah you need to rescue your own abort, a way of doing this would be:

expect { 
  expect { load_csv }.to raise_error(SystemExit)
}.to output('Muahahah').to_stderr

[myronmarston]

...or you could combined those using a compound matcher:

expect {
  load_csv
}.to raise_error(SystemExit).and output('Muahahah').to_stderr

[marko-avlijas]

Shouldn't rspec really catch SystemExit between every example?

If for no other reason, so that when example executes abort / or exit when it shouldn't, so that other examples after it execute as well.

[myronmarston]

Shouldn't rspec really catch SystemExit between every example?

If for no other reason, so that when example executes abort / or exit when it shouldn't, so that other examples after it execute as well.

We made the decision not to. This is motivated by a couple things:

• In general, we do not want to interfere with normal ruby constructs. A call to exit is supposed to make the Ruby interpreter exit and I don't see a reason for RSpec to neuter that method and make it no longer do what it is supposed to do. This is the same philosophy behind our zero-monkey patching mode.
• Even if we did rescue SystemExit -- which would neuter exit -- it wouldn't do anything about exit!. IMO, it would be confusing and inconsistent for RSpec to neuter one but not the other.

IOW, If you don't want the ruby interpreter to exit, do not call the method that is documented as making it do so. There are plenty of other ways to structure your code and/or your tests to deal with this.

[marko-avlijas]

In general, we do not want to interfere with normal ruby constructs. A call to exit is supposed to make the Ruby interpreter exit

A call to exit is meant to exit the interpreter, not to prevent test framework from running tests.

Even if we did rescue SystemExit -- which would neuter exit -- it wouldn't do anything about exit!. IMO, it would be confusing and inconsistent for RSpec to neuter one but not the other.

IMO it's far more confusing for tests to be swallowed with no explanation why.

IOW, If you don't want the ruby interpreter to exit, do not call the method that is documented as making it do so. There are plenty of other ways to structure your code and/or your tests to deal with this.

I want it to exit in my code, but I don't want it to break rspec.
Integration tests of CLI apps (end-to-end) will always have to test code which exits the program.

[myronmarston]

A call to exit is meant to exit the interpreter,

Exiting the interpreter will cause RSpec to exit because RSpec is running in the interpreter.