Tag Archives: RSpec

Precision Failure

Posted on by
Reply

When tests fail it’s nice to know why. The more precise a failure message is and the less time required to investigate why the test failed, the better.

When trying to find out what broke the test, this…

Failure/Error: search_field.should_not be_visible
       expected visible? to return false, got true

…is infinitely preferable to this…

Failure/Error: search_field.visible?.should == false
       expected: false
            got: true (using ==)

The first failure tells you that the test expected the search field to be invisible, the second that it expected true to equal false – not very helpful. I’d rather have the first error message than the second, and unless you’re a masochist you probably would like the same (though having reviewed a lot of test code, I’m not so sure…).

Cucumber and rspec make “precision failure” easy through the use of matchers. Instead of checking whether “.visible?” returns true or false, you can use rspec matchers to write the following code:

search_field.should_not be_visible
...or...
search_field.should be_visible

…instead of this:

search_field.visible?.should == false
...or...
search_field.visible?.should == true

Your test code will be more understandable, and when tests fail you’ll have a high chance of knowing exactly what went wrong.

So, go and learn about RSpec Matchers!

How to get the RSpec test result in the after block

Posted on by
Reply

—UPDATE—

The change I asked for made it into rspec so you can now call example.exception out of the box!

—/UPDATE—

RSpec allows you to run a block of code at the end of each test using the after(:each) method. A change is going to be included in a future (hopefully near future!) version of RSpec that allows you to know what exception the test failed with so that you can decide what to do based on the exception. You’ll be able to do that by asking the example for its exception:

I’ve been using a monkeypatch for a little while that allows me to know if a test passed or failed – I’ve wanted to write out a variable on test failure but until now rspec hasn’t exposed the test result. Here’s the patch:

If you place the above code in your project somewhere, you’ll be able to use it as shown in the following example of its use:

When the next version of RSpec exposes the exception that caused the test to fail, this patch will still work, so it’s fairly well future proof…

Custom RSpec ‘progress-with-names’ formatter

Posted on by
1

RSpec‘s progress formatter (the one that produces output like ......F..*...FF....) produces very concise output – which is usually all anybody wants. But there’s a problem with that: if your tests are being run from hudson/jenkins and you want to watch the progress go by on the console screen, no output is displayed… until the output generates a newline character which only happens when a test fails. Annoying.

In order to force each test result to be displayed immediately in the console window, the result needs to include a newline character after either the ‘.‘, ‘F‘, or the ‘*‘. But output like that would be ugly. Adding the test name to the output would make the output look less ridiculous and would also let us know the results of individual tests immediately instead of having to wait for all the tests to finish. So I wrote an rspec formatter to do just that – it’s basiclly a rip-off of the progress formatter. Here it is:

To use it, save it to a file called ‘progress_with_names.rb‘, put it in your working directory and invoke it as explained below…

Example output

The progress_with_names formatter produces output like the following:

nat$ rspec test_spec.rb -f ProgressWithNames
. This should pass
F This should fail
* This should be pending

…which is way more informative (and hudson/jenkins friendly) than the corresponding progress formatter output:

nat$ rspec test_spec.rb
.F*

How to use from rake

A very simple rspec rake task that uses the progress_with_names formatter:

RSpec::Core::RakeTask.new do |t|
t.pattern = '**/*_spec.rb'
t.rspec_opts = ["-r", "./progress_with_names.rb", "-f", "ProgressWithNames"]
end

Example RSpec project

I’ve included a very simple rspec project that demonstrates the use of the formatter. Download it here, and then run:

rake spec

…to see the output!

Hope that helps…

RSpec and ci_reporter

Posted on by
4

In order to use rspec within hudson, you need to use the ci_reporter gem. The gem extends rspec’s behaviour to include the junit-style output required by hudson. Annoyingly, the documentation for creating a rspec rake task that uses ci_reporter is a bit sparse and not particularly “googleable”, so I’m putting some example code here for posterity. The following is all that is required in a Rakefile for the most basic rspec task that integrates with hudson:


require 'rspec/core/rake_task'
require 'ci/reporter/rake/rspec'

RSpec::Core::RakeTask.new(:all => ["ci:setup:rspec"]) do |t|
  t.pattern = '**/*_spec.rb'
end

You can now run the rspec task named “all” within hudson and get pretty results, graphs, and all the other goodness that hudson derives from junit output files. To run the above task:

rake all

Hope that saves you some searching around!

Is Cucumber adding value to your project? The Chicken Test!

Posted on by
2

Don’t get me wrong – I love cucumber. I’ve been on projects where it has been used successfully and I’d partly attribute the success of those projects to the use of cucumber. It is a fantastic tool for BDD. When used right:

  • its scenarios provide a definitive specification of app functionality
  • its output provides one of the most useful metrics of progress: running passing tests
  • it gets everyone talking the same language
  • it provides a layer of abstraction between the required behavior and the app implementation (if you keep your steps declarative)
  • it helps to keep the focus on what code really needs to be written

All of this from one tool! Pretty good, huh? Well, like anything good, it comes with costs – those’ll be covered in a later post.

Whether the costs are outweighed by the benefits that cucumber can provide is dependent on how closely the customers, devs and testers are working together. The more blurred the roles of dev and test are and the more involved the customer is, the more value you’ll get from using cucumber. The further the devs are from the testers and the less involved the customer is, the less value cucumber will give you.

There comes a point where the costs of using cucumber outweigh the benefits – and that point comes along pretty quick IMHO. Most of the projects that I’ve seen using cucumber really shouldn’t be. Yes, it’s the flavor of the month. Yes, it’s what all the cool kidz are using. But it’s not a panacea. Using cucumber will not shower your project with pink unicorns or make your team collaborative. Too often I’ve seen cucumber used for CVDD/RDD instead of BDD – almost always a bad move…

Having been on quite a few projects that have used cucumber, I’ve come up with an unscientific and clumsy acid test that will show you if you should stop using cucumber and move to something with lower maintenance costs (eg: rspec/testunit/whatever). Here, in all its glory, is The Chicken Test™:

Instead of your usual feature description, try putting the following at the top of your next feature file:

Feature: [whatever feature is being written]
  As a chicken
  I want to cross the road
  In order to get to the other side

Possible results of The Chicken Test:

  • You get caught while typing - Pass. To have got caught this early you must be writing your scenarios collaboratively. Continue using cucumber.
  • You manage to get it checked in - I have a bad feeling about this… Get people talking again.
  • No one notices for a couple of days even though the scenarios are being run - Fail. If people aren’t even reading the scenarios, they’re not serving their purpose. Use rspec.
  • No one notices for a couple of months – Epic fail. Wipe that smug look off your face – you’ve made your point. The devs don’t want you testers “in the way”, so you may as well make life easier for yourself – move to rspec.

All too often, projects I’ve seen fail the chicken test. Does yours?

Get HTML output from Test::Unit by using RSpec !?

Posted on by
9

Test::Unit‘s lack of pretty xml/html output led me to look around to see what was available. Turns out that there’s a very simple way to get html output… run your Test::Unit tests with rspec! Here’s a contrived and simple example of how it’s done.

We’ll start with a file called test_unit_to_rspec.rb containing only two tests; one that passes and one that fails. Here’s the code:

require "rubygems"
require "test/unit"
class SomeTests < Test::Unit::TestCase
  def test_that_passes
    assert(true)
  end
  def test_that_fails
    assert(false)
  end
end

To run this, we call ruby test_unit_to_rspec.rb and we get the following output:

Loaded suite /Users/nat/dev/test_unit_to_rspec
Started
F.
Finished in 0.011906 seconds.
1) Failure:
test_that_fails:8
is not true.
2 tests, 2 assertions, 1 failures, 0 errors

A ‘.’ and an ‘F’ in the result tells us that one test passed and one failed. Notice, no pretty html. No xml. This is a sad state of affairs and needs to be fixed. We want pretty. We want red. We want green. We want output that will be useful to people other than the test team. So… to fix this…

First, you’ll need to install the syntax gem (gem install syntax). This adds extra prettiness to the output – and that’s what we’re about at the moment! Then, in our test_unit_to_rspec.rb file…

…replace :

require "test/unit"

…with:

require "spec/test/unit"

The new code will look like this:

require "rubygems"
require "spec/test/unit"
class SomeTests < Test::Unit::TestCase
  def test_that_passes
    assert(true)
  end
  def test_that_fails
    assert(false)
  end
end

Next, run your tests using the following command (notice, no ruby in the command):

spec test_unit_to_rspec.rb -f h > results.html

When the run is finished, you’ll find pretty html output in the new file named results.html; here’s what it looks like:

rspec_output

Pretty! Red! Green! Summaries! The code that failed! A whole new world of beautiful (and useful)…

Note that this is only the first step in converting to rspec from test::unit. Doing the above will give you none of the power of rspec, only the pretty output. Also, I imagine that as I get around to converting bigger test::unit projects, conversion won’t be this easy…