So, I’ve previously put together an JUnit formatter for RSpec so I can use RSpec with Jenkins/Hudson, but it was a bit of a hack. People seem to use and like it so I’ve redeveloped it and packaged it as a gem. I’ve also fixed a few problems with it, particularly how it groups tests together. It’s called… “Yet Another RSpec JUnit Formatter“, AKA: yarjuf. Inspired, I know.
Read about it here: https://github.com/natritmeyer/yarjuf
To install: gem install yarjuf
While RSpec is great, it is missing a built in JUnit formatter so you don’t get the pretty analysis stuff from Jenkins that you otherwise would. There have been a few attempts at writing an RSpec JUnit formatter, but I’ve never seen something work consistently across versions of RSpec. In my experience, the ci_reporter gem works well, but only with some versions of RSpec. Since it doesn’t work with the current version (2.10), I decided to write my own simple JUnit formatter that will allow me to run my tests in Jenkins. To use it, save the code that follows to a file called junit.rb and then call rspec with the following options:
rspec my_spec.rb -r ./junit.rb -f JUnit -o results.xml
That will cause RSpec to require the file containing the formatter, invoke the formatter and save the JUnit-formatted results to a file called results.xml. A better option would be to require the junit.rb file from your spec_helper.rb file (which in turn is being required by your .rspec file, right?). If you do that then you’ll only need to call the following:
rspec my_spec.rb -f JUnit -o results.xml
Anyway, here’s the code. Hope it helps!
Skorks, a great ruby/dev blog, gave the following challenge last year:
what is the minimum amount of code needed to make a viable unit testing framework?
An interesting question… Well, he chose to replicate rspec, and managed it in 44 lines of code. Not bad! I decided to try the same but instead of rspec, I’d try to create a minimal version of the granddaddy of them all: test::unit. Here are the tests that I’d use to write the test runner against:
And this is the kind of output I’d want the tests to produce:
So… how to make this work… in as little code as possible…
Test::unit works around the idea of test classes. Any method that begins with test_ that is defined in a class that inherits from Test::Unit::TestCase would get executed in the context of a new instance of the class. Instead of the long-winded class name from test::unit, my test class name is TinyTest. The only assertion I’m going to provide is assert which tests that its argument is true. If what’s passed is not true then TinyTest will raise an error saying which line of which file the failed assertion was on; its class and method would also be reported. I also want setup and teardown functionality.
Like Test::Unit, I don’t want to have to tell the tests to execute, I want that to happen by magic. So, all the execution logic would need to go in an at_exit block. I want a passing test to print a ‘.‘, and a failing test to print an ‘F‘. Just like Test::Unit does. Finally, at the end of executing all the tests, I want to know how many of them passed and how many failed; and for each failure I want to know the file name, the line number, the class name and the method where the failure occurred.
Here’s what I put together:
28 lines! Not bad!!! How’s about that, huh? OK, it’s no cucumber, rspec, minitest or even test::unit… but it works! It does the job of a super simple test runner! I guess I could squash it further by using semi-colons instead of new lines, but that would be cheating. 28 lines it is!
When tests fail it’s nice to know why. The more precise a failure message is and the less time required to investigate why the test failed, the better.
When trying to find out what broke the test, this…
Failure/Error: search_field.should_not be_visible
expected visible? to return false, got true
…is infinitely preferable to this…
Failure/Error: search_field.visible?.should == false
expected: false
got: true (using ==)
The first failure tells you that the test expected the search field to be invisible, the second that it expected true to equal false – not very helpful. I’d rather have the first error message than the second, and unless you’re a masochist you probably would like the same (though having reviewed a lot of test code, I’m not so sure…).
Cucumber and rspec make “precision failure” easy through the use of matchers. Instead of checking whether “.visible?” returns true or false, you can use rspec matchers to write the following code:
search_field.should_not be_visible ...or... search_field.should be_visible
…instead of this:
search_field.visible?.should == false ...or... search_field.visible?.should == true
Your test code will be more understandable, and when tests fail you’ll have a high chance of knowing exactly what went wrong.
So, go and learn about RSpec Matchers!
—UPDATE—
The change I asked for made it into rspec so you can now call example.exception out of the box!
—/UPDATE—
RSpec allows you to run a block of code at the end of each test using the after(:each) method. A change is going to be included in a future (hopefully near future!) version of RSpec that allows you to know what exception the test failed with so that you can decide what to do based on the exception. You’ll be able to do that by asking the example for its exception:
I’ve been using a monkeypatch for a little while that allows me to know if a test passed or failed – I’ve wanted to write out a variable on test failure but until now rspec hasn’t exposed the test result. Here’s the patch:
If you place the above code in your project somewhere, you’ll be able to use it as shown in the following example of its use:
When the next version of RSpec exposes the exception that caused the test to fail, this patch will still work, so it’s fairly well future proof…
RSpec‘s progress formatter (the one that produces output like ......F..*...FF....) produces very concise output – which is usually all anybody wants. But there’s a problem with that: if your tests are being run from hudson/jenkins and you want to watch the progress go by on the console screen, no output is displayed… until the output generates a newline character which only happens when a test fails. Annoying.
In order to force each test result to be displayed immediately in the console window, the result needs to include a newline character after either the ‘.‘, ‘F‘, or the ‘*‘. But output like that would be ugly. Adding the test name to the output would make the output look less ridiculous and would also let us know the results of individual tests immediately instead of having to wait for all the tests to finish. So I wrote an rspec formatter to do just that – it’s basiclly a rip-off of the progress formatter. Here it is:
To use it, save it to a file called ‘progress_with_names.rb‘, put it in your working directory and invoke it as explained below…
Example output
The progress_with_names formatter produces output like the following:
nat$ rspec test_spec.rb -f ProgressWithNames
. This should pass
F This should fail
* This should be pending
…which is way more informative (and hudson/jenkins friendly) than the corresponding progress formatter output:
nat$ rspec test_spec.rb
.F*
How to use from rake
A very simple rspec rake task that uses the progress_with_names formatter:
RSpec::Core::RakeTask.new do |t|
t.pattern = '**/*_spec.rb'
t.rspec_opts = ["-r", "./progress_with_names.rb", "-f", "ProgressWithNames"]
end
Example RSpec project
I’ve included a very simple rspec project that demonstrates the use of the formatter. Download it here, and then run:
rake spec
…to see the output!
Hope that helps…
UPDATE: You may find this RSpec JUnit Formatter to be less hassle/more useful than ci_reporter…
In order to use rspec within hudson, you need to use the ci_reporter gem. The gem extends rspec’s behaviour to include the junit-style output required by hudson. Annoyingly, the documentation for creating a rspec rake task that uses ci_reporter is a bit sparse and not particularly “googleable”, so I’m putting some example code here for posterity. The following is all that is required in a Rakefile for the most basic rspec task that integrates with hudson:
require 'rspec/core/rake_task'
require 'ci/reporter/rake/rspec'
RSpec::Core::RakeTask.new(:all => ["ci:setup:rspec"]) do |t|
t.pattern = '**/*_spec.rb'
end
You can now run the rspec task named “all” within hudson and get pretty results, graphs, and all the other goodness that hudson derives from junit output files. To run the above task:
rake all
Hope that saves you some searching around!
If you’re upgrading from rspec 1 to rspec 2, details of the differences are here:
http://github.com/rspec/rspec-core/blob/master/Upgrade.markdown
The specific changes you’ll need to make to your rake file can be found here: