Tag Archives: agile testing

RSpec JUnit Formatter for Jenkins

Posted on by
11

While RSpec is great, it is missing a built in JUnit formatter so you don’t get the pretty analysis stuff from Jenkins that you otherwise would. There have been a few attempts at writing an RSpec JUnit formatter, but I’ve never seen something work consistently across versions of RSpec. In my experience, the ci_reporter gem works well, but only with some versions of RSpec. Since it doesn’t work with the current version (2.10), I decided to write my own simple JUnit formatter that will allow me to run my tests in Jenkins. To use it, save the code that follows to a file called junit.rb and then call rspec with the following options:

rspec my_spec.rb -r ./junit.rb -f JUnit -o results.xml

That will cause RSpec to require the file containing the formatter, invoke the formatter and save the JUnit-formatted results to a file called results.xml. A better option would be to require the junit.rb file from your spec_helper.rb file (which in turn is being required by your .rspec file, right?). If you do that then you’ll only need to call the following:

rspec my_spec.rb -f JUnit -o results.xml

Anyway, here’s the code. Hope it helps!

Is Cucumber adding value to your project? The Chicken Test!

Posted on by
2

Don’t get me wrong – I love cucumber. I’ve been on projects where it has been used successfully and I’d partly attribute the success of those projects to the use of cucumber. It is a fantastic tool for BDD. When used right:

  • its scenarios provide a definitive specification of app functionality
  • its output provides one of the most useful metrics of progress: running passing tests
  • it gets everyone talking the same language
  • it provides a layer of abstraction between the required behavior and the app implementation (if you keep your steps declarative)
  • it helps to keep the focus on what code really needs to be written

All of this from one tool! Pretty good, huh? Well, like anything good, it comes with costs – those’ll be covered in a later post.

Whether the costs are outweighed by the benefits that cucumber can provide is dependent on how closely the customers, devs and testers are working together. The more blurred the roles of dev and test are and the more involved the customer is, the more value you’ll get from using cucumber. The further the devs are from the testers and the less involved the customer is, the less value cucumber will give you.

There comes a point where the costs of using cucumber outweigh the benefits – and that point comes along pretty quick IMHO. Most of the projects that I’ve seen using cucumber really shouldn’t be. Yes, it’s the flavor of the month. Yes, it’s what all the cool kidz are using. But it’s not a panacea. Using cucumber will not shower your project with pink unicorns or make your team collaborative. Too often I’ve seen cucumber used for CVDD/RDD instead of BDD – almost always a bad move…

Having been on quite a few projects that have used cucumber, I’ve come up with an unscientific and clumsy acid test that will show you if you should stop using cucumber and move to something with lower maintenance costs (eg: rspec/testunit/whatever). Here, in all its glory, is The Chicken Test™:

Instead of your usual feature description, try putting the following at the top of your next feature file:

Feature: [whatever feature is being written]
  As a chicken
  I want to cross the road
  In order to get to the other side

Possible results of The Chicken Test:

  • You get caught while typing - Pass. To have got caught this early you must be writing your scenarios collaboratively. Continue using cucumber.
  • You manage to get it checked in - I have a bad feeling about this… Get people talking again.
  • No one notices for a couple of days even though the scenarios are being run - Fail. If people aren’t even reading the scenarios, they’re not serving their purpose. Use rspec.
  • No one notices for a couple of months – Epic fail. Wipe that smug look off your face – you’ve made your point. The devs don’t want you testers “in the way”, so you may as well make life easier for yourself – move to rspec.

All too often, projects I’ve seen fail the chicken test. Does yours?

ThoughtWorks Anthology: Agile vs Waterfall Testing

Posted on by
Reply

A pragprog book by the title “ThoughtWorks Anthology – Essays on Software, Technology and Innovation” has been hanging around the office gathering dust for the past few months. While waiting for a regression test run to finish today, I picked up the book and found, on page 177, a chapter (no 13) with the title: “Agile vs Waterfall Testing for Enterprise Web Apps”. Intrigued, I borrowed the book and read the chapter on the way home.

ThoughtWorks Anthology

If you haven’t done agile testing before or have just started and want a gentle introduction to the differences between testing in a waterfall world and the agile world, this is a great book.

It goes through the following:

  • Comparison of the waterfall and agile Testing Lifecycles
  • The different types of testing that occur (unit, functional, exploratory, etc)
  • Environment management (dev vs int vs stage environments) – what kind of testing to do where; what kind of sign-off to get in which environment
  • Tools required to get the job done (…though what is QTP doing in a list of recommended software automation tools!? …in an agile-focused book!?!!? …seriously?!?!???!)
  • Test-related roles within the team
  • …a few more things

Seriously, if you want a good high-level intro to agile testing, get this book. If you’ve been doing agile testing for a while, it’s still worth skimming over.

Quick and easy Watir test suites with Test::Unit

Posted on by
3

A test team usually has a need for a few fixed test suites, eg: a sanity suite or a suite which contains all tests for a full run. There’s usually also a need to be able to create suites with arbitrary tests in it, eg: a suite that tests all account management functionality or a suite that runs all tests for a particular platform. There is often the need to quickly throw together a suite which can be used to regression-test a specific area of functionality.

If you’re using a ruby + watir + test::unit framework, there’s a simple way to get all of the above flexible suite buliding functionality. Here’s how…

First of all, a few prerequisites… They are:

  1. All your TestCase class files should be in one folder
  2. Your TestCase class file names should follow a strict and scalable naming convention

For the purposes of this blog post, we’ll use a simple naming convention: all test files in our imaginary project begin with test_ , all suite files begin with suite_ and all test files that test account functionality contain the word account in their file name.

Test::Unit has the concept of ‘require files’… any file that requires a TestCase file will cause the test to be executed (unless something has been done to prevent tests from being run). These require files can be used as test suites – any tests that are ‘required’ in the test suite will get executed! So, the first way you could use these require files is to individually require each TestCase file. Here’s an example… a file called suite_account_tests.rb containing the following lines; each one will load a file containing a TestCase file:

require "test_account_join"
require "test_account_close"
require "test_account_upgrade"
require "test_account_change_details"

This works, but it is laborious to build, and even more of a pain to maintain. There has to be a better way… and here it is: instead of requiring each individual file that has the word account in it, you can get ruby to do the work for you. Replace the contents of suite_account_tests.rb with the following line:

Dir.glob(File.join(File.dirname(__FILE__), '*account*.rb')).each {|f| require f }

That one line will read in all ruby files that contain the word ‘account’ in the file name, and that live in the same directory as the suite_account_tests.rb file. Then, because this is just a test::unit require file, all the TestCase classes that got required will get executed. Awesome. No need to update the suite file if an account-related TestCase file is added or deleted; it will pick up any changes automatically. All that’s required is a good naming convention…

The power of that one-liner lies in the Dir.glob function. It takes filename ‘patterns’ (shame it doesn’t take a regex) documented in the Dir.glob rdoc to decide which files to run. If you’ve got a strict naming convention, you’ll find that the Dir.glob functionality lets you create suites very quickly. If you can’t create suites quickly now, you’ll find that it’s great PR for the test team when you can!

Using these patterns, with the following one-liner we can build a suite that will run all tests:

Dir.glob(File.join(File.dirname(__FILE__), 'test_*.rb')).each {|f| require f }

All files that begin with test_ will get required and executed. This works because of the file naming convention we’re using. Only TestCase files begin with test_, so we can be sure that only tests are getting loaded. Specifically, suite files won’t be loaded as their file names begin with suite_, not test_. Anyway, you get the idea.

There’s nothing stopping you from combining the two approaches. You can have a require file that uses both the Dir.glob approach that also has individual requires if the suite needs to include specific TestCases classes.

One non-obvious advantage that this Dir.glob approach to suite files gives you is that you won’t ‘lose’ tests any more. I’ve found that when I’ve used the individual-require approach, on occasion I forget to add a require to the suite when I create a new TestClass file. I end up with tests that gather dust – they never get run. They are forgotten and left to rot. By the time I find them again, the tests are so out of date that they often need rewriting, never mind editing!

Ruby gives you a whole load of power for free; you may as well use it!

Fitnesse, Ruby and the Mac

Posted on by
Reply

I’ve recently started using Fitnesse, an acceptance testing framework. There really isn’t much around on how to get it working on the mac with ruby, so I decided to get it working and put up my findings…

Installing the Fit Ruby Gem

First of all, you’ll need to download the Fitnesse ruby gem. It isn’t complete by any stretch of the imagination, but it does the job. To get it, run the following command, entering your administrator password when prompted:


sudo gem install fit

You should see the following as a result:


Successfully installed fit-1.1

Installing the Fitnesse Server

Now, you’ll need to download Fitness… Click here, and select the most recent version (20070619 at time of writing). I downloaded the fitnesse20070619.zip version.

Unzip the file, and copy the resulting fitnesse folder to the Applications folder (there’s where I put it… if you do the same, it’ll be easy to follow along with this post).

Allowing Fitnesse to run

We now have to set the correct unix file permissions to allow the Fitnesse server to run. Back to the terminal, this time entering the following commands:


cd /Applications/fitnesse/
chmod +x run.sh

That sets the “I can be executed” flag on the file, allowing us to run the server. Not very Mac-like, but it has to be done.

Setting up the Ruby Fixture Code

We’ll now prepare the ruby fixture code (ie: the test) that we’ll eventually be executing. Having this in place before we create the Fitnesse page gives us the ability to test everything we do whilst trying to get it all to work – it’s a painful process, anything to make it easier is a good thing).

I created the following file…

/Users/nat/Development/Projects/Ruby/WatirFitFramework/lib/Framework/Testcase.rb

…with the following contents…

require 'rubygems'
require 'fit/column_fixture'
module Framework
class Testcase < Fit::ColumnFixture
attr_accessor :numerator
attr_accessor :denominator
def quotient
@numerator.to_f / @denominator.to_f
end
end
end

Make sure that the the case used for the name is consistent. I advise using a filename beginning with a capital letter, followed by only lowercase letters.

Setting up the Fitnesse Test Page

We first of all need to start the fitnesse server. Back to the terminal…


cd /Applications/fitnesse/
./run.sh -p 8080

The “-p 8080″ means that the fitnesse server will be running on port 8080. Ports lower than 1024 will cause errors.

Now open Safari (or any other web browser – I’m using safari for this post) and navigate to http://127.0.0.1:8080. If you see the “Welcome to Fitnesse!” page, all is good.

Due to the way that test suites work in fitnesse, I decided to put the config stuff in a parent page, and the test itself in a child page. I haven’t got the space/time/energy to explain that concept – see the fitnesse documentation for that.

I created the “NatTest” page by going to http://127.0.0.1:8080/NatTest and clicking on the “create this page” link that is displayed as a result.

In the page (click the ‘Edit’ link in the navigation column) I added the following:


!define COMMAND_PATTERN {ruby -I %p /Library/Ruby/Gems/1.8/gems/fit-1.1/bin/FitServer.rb}
!path /Users/nat/Development/Projects/Ruby/WatirFitFramework/lib

The first line tells fitnesse how to run ruby, and the second line tells fitnesse where our ruby fixture code is.

Next, I created the actual fitnesse test by navigating to http://127.0.0.1:8080/NatTest.TheTest, again clicking on the “create this page” link. In it, I pasted the following ColumnFixture table:


|Framework.Testcase|
|numerator|denominator|quotient?|
|10|2|5|
|12.6|3|4.2|
|100|4|24|

Last thing now… we need to make our “TheTest” page an actual test so that fitnesse can execute it. To do that, click on the “Properties” link in the navigation column, check the “Test” check button and then click “Save properties”.

Run the test

In the navigation column there should now be a “Test” button. Click it…

If all has gone well, you should see some test results. From here on, you’re by yourself.