Category Archives: agile

SitePrism: Capybara Page Objects

Posted on by
3

Capybara is a great browser interaction library for automated testing, but until now it hasn’t been much fun to follow the Page Object pattern with it. So, I wrote SitePrism that lets you do just that: SitePrism is a Page Object Model DSL for Capybara.

We’ve been using it at my current client since December 2011 and it’s proven a great success. Until now I’ve kept it quiet to allow some time to develop it to the point that it does everything we need, but a few days ago SitePrism went to version 1.0 – early adopters tell me that SitePrism’s intuitive API lets them create expressive and maintainable Page Objects with ease, so take a look! The SitePrism ReadMe will give you an idea of what it’s like to put together a Page Object model of the site you’re testing using SitePrism with capybara…

Here’s the code: https://github.com/natritmeyer/site_prism
And here’s the documentation: http://rdoc.info/gems/site_prism/frames

Go take a look, it might save you a lot of frustration!

Step-by-step example of BDD’ing a WPF app with Cucumber, RSpec, IronRuby and Bewildr

Posted on by
6

In this tutorial, we’ll go through the BDD cycle step-by-step and develop a small Microsoft WPF app to solve a contrived problem. We’ll go through how to specify the desired features in cucumber, how to implement the steps in ironruby using bewildr to automate the UI, how to use rspec from ironruby to do the unit testing, we’ll write the C# code to pass the unit tests, and the WPF to create a UI to pass the cucumber scenarios! Get comfortable – this may take some time…

Things I’m going to assume

  • You know enough C# to write a simple WPF GUI
  • You know enough ruby, rspec and cucumber to automate some basic tests
  • You have a reasonable grasp of the concepts behind BDD and TDD
  • The following code will all be done in c:\wpfbdd

Ingredients

You will need the following installed…

With that, we are ready to BDD some WPF!

Introducing the Problem
Software exists to solve problems. What’s the problem that our contrived little app needs to solve? It’s time to jump into tutorial-scenario-land where everything is nice’n'easy and problems are well defined…

Farmer Giles has a flock of sheep that he moves from field to field. If any of the flock get left behind, they’ll get eaten by the Big Bad Wolf, which Farmer Giles would find sad and costly. What he wants is some way to keep track of his sheep as he moves them all from one field to the next to make sure that he hasn’t left any stragglers behind.

It’s time for a cucumber feature file
Hmmm… that’s enough material to start with. It’s time to begin writing our feature file. Here it is:

Save that to features\sheep_counter.feature. We have a conversation with Farmer Giles and he’s happy that we seem to have a general idea of what he wants. Next, we need to add some scenarios to our feature file. After a bit more chatting with Giles we decide that what he wants is a sheep counter, and these are the scenarios we come up with:

OK, now we’re getting somewhere. Giles is happy for us to build a sheep counter and we’ve got a few scenarios of how he’d like to use it… it’s time to start writing the step definitions!

Filling in the Step Definitions
Before we fill in the step definitions, we’ll quickly create the features\support\env.rb file with the following contents:

Now that bewildr has been added to the project, we can go ahead and create the features\step_definitions\sheep_counter_steps.rb file with the following contents:

If you’re struggling to keep up, those are the code blocks that will get called when you run the icucumber command (NB: that’s icucumber, not cucumber). Before you run icucumber, take the time to bask in the beauty that is the bewildr API displayed in the above steps… Enjoyed that? Good. Bask a bit more. And again. Hmmm… that’ll probably do. Back to work. Run the icucumber command, and take a look at the output:

Ugly. There’s a theme in the ugliness though. Here is the problem that’s causing the scenarios to fail:

Can't find: c:\wpfbdd\SheepCounter\SheepCounter\bin\Debug\SheepCounter.exe (RuntimeError)

That’s an error from bewildr telling us that it can’t find SheepCounter.exe. Well, that’s no big surprise given that it doesn’t exist! We now need to start writing some code to make these scenarios go green.

Our first bit of production ‘stuff’!
It’s time to break open Visual Studio and create a new project of type “WPF Application”. Name the project “SheepCounter” and save it to c:\wpfbdd. You’ll be greeted with a window something like this:

We’ll get to the code in a minute, just “Start Debugging” the new WPF project. That should be enough to compile it and get a blank WPF window displayed. Close the WPF window and go back to your command prompt and give icucumber another go. This time, you’ll see that the scenarios are opening the app, but then fail with ElementDoesntExist errors. Here’s the console output you should be getting now:

So, the scenarios are still all failing, but at least our new SheepCounter.exe is opening. But… it’s opening one instance per test and leaving them around. We need to change things so that the app gets closed at the end of the test. We need to update our features\support\env.rb file to include an After block:

If you run icucumber again you’ll notice that though the scenarios still fail, at least the SheepCounter app is being tidied up at the end of each test.

We’ll do a final bit of tidying up ourselves and rename our WPF window from “MainWindow” to “Sheep Counter” (it’s the window’s “Title” property that you need to change). When you’ve done that, run the WPF app again to verify that the window name has changed.

Now that the WPF app’s window has the correct title, the first of our step definitions should pass. Run icucumber again and you should see the following console output:

Better! The first step of each scenario is now passing.

This is where BDD/TDD really shine. You decide what you want your app to do, you express the desired behaviour in scenarios and then implement just enough production stuff to make the scenarios pass. When they do, you can stop coding in the knowledge that your software behaves the way you wanted it to!

First attempt at implementing the UI
From executing our scenarios we know that the app is starting up. But as soon as it starts, the scenarios fail because they can’t find the various UI elements that are mentioned in the step definitions. We need to go through the step definitions and add any UI elements they mention to the WPF UI.

As a minimum you need to add a label whose name is ‘sheep_count’ and a button whose name is ‘increment_sheep’. Here’s what I’ve got at this point (note, I’ve given the button some content and I’ve got an extra label with the contents “Sheep Count:”, just to make the app a litte more pretty and little less utilitarian):

Run the WPF app again (in debug mode) to make sure that all the elements are there (this will also compile the app so that the scenarios are dealing with the latest changes). Now kill the app and run icucumber again…

… the first two scenarios will run (and fail) but on the third test, it’ll get stuck in an infinite loop – the step is expecting the sheep count to increment every time the increment button is clicked, but we haven’t implemented that yet. Simply close the WPF window and the scenario will end. Your console output should now be something like:

Progress! We are now getting logic errors rather than UI errors, eg: the scenarios are complaining that they were expecting the label to contain “0″, but it got “Label” instead. When you get to this sort of point, it’s usually time to start writing some unit tests…

A Rake intermission…
Just before we get started with unit testing, we’ll add a Rakefile to our project. Rake is a great tool for managing project tasks, and since we’re going to be adding another one when we add unit tests, we may as well involve rake right now. Add a file called ‘Rakefile’ in your project root with the following contents:

All done. Now, instead of running icucumber to run our acceptance tests, we run irake cucumber. The task we’ll use to run our unit tests is irake spec.

Some unit tests…
It’s time to break out rspec and use it to design an object that can keep track of sheep for us – we of course don’t want this logic in the UI code, we need to separate it out. We need to create a c:\wpfbdd\spec directory to contain our unit tests. Once done, add a file named spec\counter_spec.rb. Now go ahead and run irake spec and you should get the following output:

It’s just telling us that we have no tests.

So what are we designing with our unit tests? The class that will manage the count of sheep as they move from field to field. Since we’re in tutorial-scenario-land, this is nice and easy. We need a class that basically keeps track of a count. It needs to start at 0, it needs to increment. That’s pretty much it for now. So we’ll start by adding a few pending tests to our spec file:

When you run that, you’ll see mentions of pending tests:

Next, we’ll write the first test:

…and when we run that with irake spec we get the following output:

So we’re finally at the stage where the test is failing for a proper reason – the dll that we want to test doesn’t yet exist. To get things working, we need to…

Add a Counter project in VS
We now need to fire up Visual Studio, and add a new ‘Class Library’ project to the solution; we’ll call the project ‘Counter’. The pre-defined class in the new Counter project is called ‘Class1.cs’. Rename that to ‘Counter.cs’. You should see something like this:

If you debug the project (to compile our new class) and then run the tests, you should now get the following obtuse error:

The key bit of info from that fail is the following: undefined method `count' for Counter.Counter:Counter::Counter. The class that we’re testing needs a count() method. So let’s add one:

Now, after debugging (to compile the new code), run the tests, and you should get:

That’s our first test passing! The C# code we added now returns the count, and starts the count at 0 when the Counter.cs class is initialized.

It’s time to move on to our next test. We now need to be able to increment the count, so we’ll add the following test code:

…and if we run that, we get the following:

…which tells us that we need an increment method. We’ll update the Counter.cs with the following:

…and after debugging the code, the tests will produce the following output:

See those 2 dots? Both our tests are passing!!! We now have the confidence that our Counter.cs class does enough to power the SheepCounter UI!

Wiring up the Counter class to the WPF UI
Before dealing with the UI, we need to add a project reference from the ‘SheepCounter’ WPF project to the ‘Counter’ Class Library project. Your Solution Explorer should look something like this:

Now that the UI project knows about the Counter class, we can update the ‘MainWindow.xaml.cs’ file with the following:

If you Debug the WPF app now, you should see that when it starts, the initial count is 0. If you run the cucumber acceptance tests (irake cucumber), you’ll find that the first test now passes! (Don’t forget about that last test that get stuck in an infinite loop – when it starts, kill it).

The test output will be something like:

The first test is now passing. We’ll move onto the second test; the one that tests the increment behaviour. To do that, first go to the WPF window layout, and double click the ‘Increment’ button you added to the UI. You should now see that the MainWindow.xaml.cs file has had an ‘increment_sheep_Click()’ method added to it. Save the project and then update the ‘MainWindow.xaml.cs’ file with the following:

Debug the project to compile it, and then run the acceptance tests (irake cucumber). You should see that all of the acceptance tests now pass!!!

The customer rejoices!!!
We find Farmer Giles, and show him our shiny new SheepCounter app. He’s very pleased, and tells us that he’ll go and use it for a few days and see how it works in the real world.

Feature request…
Farmer Giles comes back from the fields and tells us, “It’s great, this sheep counter, but there’s one annoying thing about it… Once I’ve moved all my sheep from one field to another, I have to restart the app to get the count back to 0. Could you add a feature that resets the count somehow?” “Hmmm… no problem… give us a few minutes…”

We need to add a new scenario to the feature file, update the UI to include a ‘Reset’ button, add a ‘reset()’ method to our Counter class and wire up the UI. Here goes:

Expanding the feature file
We add the following scenario to our existing ‘sheep_counter.feature’ file…

…and the following to the ‘sheep_counter_steps.rb’ file…

…and run our tests, we get the following:

Our old tests pass, but the new one fails telling us that there isn’t a reset button. So let’s add one!

Adding a Reset button
Back in VS, add a button named ‘reset’ to the UI:

After debugging the app, the new acceptance test should fail, but this time with a different problem:

… it got the wrong result. Though this time the button was found and clicked, the number didn’t get reset. We need to update our Counter class to be able to reset.

Updating the Counter class
Before updating the class, we need to add a unit test. Update the counter_spec.rb file with the following test:

Running the unit tests will give us the following result:

So there’s no ‘reset()’ method on our Counter class. Let’s add the following method to the Counter class…

…run the debugger and then run our unit tests again:

Good. All out unit tests are passing, including the new one for the reset() function.

Wiring up the reset button on the UI
The final thing we need to do is to wire up the Reset button on our WPF UI to the reset() method on the Counter class. To do this, double click on the reset button in the UI. You should see that a reset_Click() method has been added to the MainWindow.xaml.cs file. Save the project and then change the reset_Click() method to look like the following:

After debugging the update WPF app, run the acceptance tests again. You should see the following output:

Nice! Our new feature is working, and we haven’t broken anything – we know that because our other tests passed!

One happy customer!
And with that, we got Farmer Giles to spend a few days in the field with his updated SheepCounter app. He loved it – he could count his sheep as they moved from field to field, he could reset the count when he was done, and the big bad wolf died from hunger. Success!

Conclusion
I hope the above worked example, though contrived, has shown that BDD’ing a WPF app is not a big deal. The tools are easy-to-use, stable, popular and free (ironruby/cucumber/rspec/bewildr). The BDD process, though a source of contention around the interwebs, is at its core quite simple – it’s about conversations, specifying expected behaviour, writing tests first and short feedback loops. If you’re not doing it already, give BDD a go.

  • It takes a lot of the uncertainty out of coding – blind alleys are rarely ventured down because the customer is forced, by being involved in the development process – to think carefully about their requirements
  • You will make peace with your QA people – if you’re interested in the quality of your code then you’re on the same side
  • Your customer will love you – they’re getting what they need rather than what they thought they wanted
  • Maintenance will no longer be a terrifying ordeal – you’ll be happy to change code because your tests will tell you if you’ve broken anything
  • I could go on, but this isn’t the place really…

Anyway, I hope that helps!

Download the project
Click here to download the project.

Is Cucumber adding value to your project? The Chicken Test!

Posted on by
2

Don’t get me wrong – I love cucumber. I’ve been on projects where it has been used successfully and I’d partly attribute the success of those projects to the use of cucumber. It is a fantastic tool for BDD. When used right:

  • its scenarios provide a definitive specification of app functionality
  • its output provides one of the most useful metrics of progress: running passing tests
  • it gets everyone talking the same language
  • it provides a layer of abstraction between the required behavior and the app implementation (if you keep your steps declarative)
  • it helps to keep the focus on what code really needs to be written

All of this from one tool! Pretty good, huh? Well, like anything good, it comes with costs – those’ll be covered in a later post.

Whether the costs are outweighed by the benefits that cucumber can provide is dependent on how closely the customers, devs and testers are working together. The more blurred the roles of dev and test are and the more involved the customer is, the more value you’ll get from using cucumber. The further the devs are from the testers and the less involved the customer is, the less value cucumber will give you.

There comes a point where the costs of using cucumber outweigh the benefits – and that point comes along pretty quick IMHO. Most of the projects that I’ve seen using cucumber really shouldn’t be. Yes, it’s the flavor of the month. Yes, it’s what all the cool kidz are using. But it’s not a panacea. Using cucumber will not shower your project with pink unicorns or make your team collaborative. Too often I’ve seen cucumber used for CVDD/RDD instead of BDD – almost always a bad move…

Having been on quite a few projects that have used cucumber, I’ve come up with an unscientific and clumsy acid test that will show you if you should stop using cucumber and move to something with lower maintenance costs (eg: rspec/testunit/whatever). Here, in all its glory, is The Chicken Test™:

Instead of your usual feature description, try putting the following at the top of your next feature file:

Feature: [whatever feature is being written]
  As a chicken
  I want to cross the road
  In order to get to the other side

Possible results of The Chicken Test:

  • You get caught while typing - Pass. To have got caught this early you must be writing your scenarios collaboratively. Continue using cucumber.
  • You manage to get it checked in - I have a bad feeling about this… Get people talking again.
  • No one notices for a couple of days even though the scenarios are being run - Fail. If people aren’t even reading the scenarios, they’re not serving their purpose. Use rspec.
  • No one notices for a couple of months – Epic fail. Wipe that smug look off your face – you’ve made your point. The devs don’t want you testers “in the way”, so you may as well make life easier for yourself – move to rspec.

All too often, projects I’ve seen fail the chicken test. Does yours?

ThoughtWorks Anthology: Agile vs Waterfall Testing

Posted on by
Reply

A pragprog book by the title “ThoughtWorks Anthology – Essays on Software, Technology and Innovation” has been hanging around the office gathering dust for the past few months. While waiting for a regression test run to finish today, I picked up the book and found, on page 177, a chapter (no 13) with the title: “Agile vs Waterfall Testing for Enterprise Web Apps”. Intrigued, I borrowed the book and read the chapter on the way home.

ThoughtWorks Anthology

If you haven’t done agile testing before or have just started and want a gentle introduction to the differences between testing in a waterfall world and the agile world, this is a great book.

It goes through the following:

  • Comparison of the waterfall and agile Testing Lifecycles
  • The different types of testing that occur (unit, functional, exploratory, etc)
  • Environment management (dev vs int vs stage environments) – what kind of testing to do where; what kind of sign-off to get in which environment
  • Tools required to get the job done (…though what is QTP doing in a list of recommended software automation tools!? …in an agile-focused book!?!!? …seriously?!?!???!)
  • Test-related roles within the team
  • …a few more things

Seriously, if you want a good high-level intro to agile testing, get this book. If you’ve been doing agile testing for a while, it’s still worth skimming over.

Lessons from a watir success story

Posted on by
11

There aren’t enough UI test automation success stories documented on the net, so here’s my contribution.

This post is about  how we started with zero automated UI testing capability, and how a couple of months later we had the capability to run 1,000 tests in 20 minutes (you may also notice some gorilla-style chest-beating). Not only that, our test execution results were always spot on (no false passes, no false fails), and our framework was easy to maintain and extend. Here’s how we did it and the lessons we learned along the way.

The Phoenix rises from the ashes…

After several false starts with UI testing (involving QTP and watin and fitnesse and and and…), it finally got off the ground after I demonstrated what could be achieved with watir. This was the first lesson for me: demonstrating working code makes for a far better case than “just believe me, I’ve done it before, it’ll work fine”. Realizing this, I spent a weekend putting together a simple ruby/test::unit/watir testing framework.

When, on monday morning, I demoed a bunch of working tests that could run over and over and over and each time give the correct results to a room full of doubters, it was decided that we’d give ruby/test::unit/watir a go.

An important constraint

In order to avoid the biggest mistake made in previous attempts at UI automation, I insisted on agreement to a very important constraint: the ruby/test::unit/watir framework would only test the web app part of the system. One of my philosophies when it comes to UI automation is: “use the right tool for the right job”. There would be no scope creep. The framework would be used for the web app only. The WPF fat client could be tested with something else – anything else, just please don’t subvert ruby/test::unit/watir for testing anything else but the web app.

This was all part of expectations management; over the years I’ve learned that this is one of the most important things when doing automated UI testing. It’s worth a separate post – I’ll get around to it.

Framework overview

The framework I came up with that weekend was very simple (and very unoriginal – I’ve done similar stuff before): each page in the app would be represented by a ruby module. Each “page module” would contain one method per control on the web page; the method when called would return the required object. Eg:

module HomePage
  def home_page_sign_in_link
    @browser.link(:xpath, "//a[contains(@href,'signin')]")
  end
  def home_join_button
    @browser.button(:id, "join")
  end
end

My experience is that IDs on objects often can’t be trusted. What I’ve found is that they usually get put in during development and that they are removed when they’re not required. If I write my object recognition code to depend on an ID I usually regret it. I’ve learned that xpath is often more reliable – IDs tend to disappear. Where I have reliable IDs, I’ll use them. But for anything else, I use xpath. I find myself maintaining ID-based object recognition a lot more than xpath-based object recognition. Your mileage may vary.

Each of these page modules was “mixed in” to a common module which was then mixed into the test, making each object in the system available to the test without having to deal with namespacing. Not the prettiest solution, but it got us working very quickly. We surprised ourselves by having no method name clashes – mainly due to the object naming convention we used.

Our app uses restful services so the net::http libraries were great. The calls were wrapped up in classes and placed alongside our test-data-generation code (random strings, valid usernames, etc). We wrote them to be independent of the rest of the framework (apart from the central config file that pointed them at the right environment) so that we could use them whenever and however we liked. This philosophy was applied to as much of the framework as possible – to the greatest extent possible we kept everything independent and modular. This allowed us to keep up with sweeping changes in the app we were testing. We had a lot of sweeping changes; to many people’s amazement (not least our own!) we were able to keep up with everything thrown at us.

One of watir’s greatest features which doesn’t get nearly enough air time is the “checkers” functionality. Every time a page is visited you can run custom code you’ve wrapped up in a ruby proc. Environment instability usually manifests itself in obvious ways. Watir checkers allow you to recognize these as soon as they appear. If, in your checker code, you raise an exception with a fixed message, you can find these when the test is done and mark the tests as “failed due to environment instability”; or you can be more specific, e.g: “failed due to database connection going down, again”. This saves us hours of work every day. Literally, hours.

Our test execution script keeps note of tests that failed. When all the tests have been run, the tests that failed get run again in the hope that environment instability “got better”. We’d have to rerun those tests manually at the end of a run anyway, so why not automate it? A big time saver.

Test::Unit

We used test::unit for executing the tests. It’s simplicity and rock-solid nature made it a good choice to start the framework with. It is also easily expanded/monkey-patched – we added a few things to it.

Though I have ruby experience, everyone else on the team had a C#/NUnit background; test::unit helped bridge their knowledge gap. Conceptually, test::unit and NUnit are identical (both are xunit tools) – the similarity meant one less thing for the others to learn. When under pressure, that was a good thing.

Netbeans as an IDE

Netbeans‘ integration with test::unit is great, so we decided on using it for our IDE. It turns out that NetBeans is a great IDE for watir testing; I heartily recommend it. The green ticks and red crosses were comforting to all the right people, the ease of viewing the source code of the watir/firewatir libraries was great, its intellisense wasn’t as bad as I was expecting it to be and its svn integration is better than anything else I’ve used so far. It’s also free so I didn’t have to get approval to buy it! It has loads of useful plugins too.

Firefox plugins

We used firefox (and therefore firewatir) because we needed to spoof useragents and headers – using firefox profiles meant we could test the mobile, browser and fat client webstores. Using firefox meant we could use plugins; here are the ones we found most useful:

  • Firebug – it is awesome. Install it, learn it, love it.
  • XPath Checker – It’s a great tool which does one thing very well: Write an xpath and it’ll show you what it evaluates to. There are other tools that do the same thing and more but the experience of everyone on the team was that xpath viewer was all they needed. Everything else was bloat.

Version Control

One area where testing is usually decades behind development is in the use of version control. This is probably due to the fact that most commercial tools make it less than trivial to do (the QTP and RFT test file formats are hardly version-control-friendly).

One of the many joys of using watir for UI testing is that using version control is simple – there are no ‘project files’ to worry about, no accompanying files that you need to check in when you change a test file, no bizarre links across directories, no weird inter-file dependancies.

Using version control for our tests and framework meant that we were free to change stuff as we saw fit – if something didn’t work we could just revert. I can’t recommend using version control enough. The “blame” feature in svn was particularly useful ;)

Parallelization of test execution

We got our super-fast execution time by cheating: we parallelized the execution of the tests. We used a Mac Pro (16GB RAM, 8 core, 4xHDD), the Parallels virtualization software to run 12 VMs, and a shonky “n mod 12″ function to decide what tests to run on each VM (we tried to get testjour working but it needs a fair bit more dev and documentation before it works as advertised and becomes usable).

One advantage of using a Mac (apart from its rock-solid stability) was “Exposé” – we could see all 12 VMs running at once. The amount of people who walked past our desk and got hypnotized by seeing 12 tiny PCs each executing as if they were self aware… – good PR for our team!

Results sanitization

It doesn’t matter if you can run a million tests in 5 minutes, if the results aren’t accurate and sanitized quickly. We’re quite proud of the fact that we provide our results within 5 minutes of finishing the last test (usually straight away). We’ve put a lot of effort into being able to do this and it’s one of the biggest selling points of the framework we’ve written. Here’s what we’ve done to get to this stage:

  1. Though we use many VMs to execute tests on, when execution is finished we write all the results to a central location. We don’t have to worry about collating results, it’s done automatically.
  2. We use the watir checker feature heavily (see above). Any test that failed because of an environment problem is marked as such. This means that we don’t need to investigate why the test failed – we know that it didn’t get past the login screen because we saw the “database connection down” message, not because there’s a bug in login.
  3. Where we find a bug, we put the number in the relevant assertion message. If the test fails at the same point in the next run, it’s because the bug is still there. The bug will be printed to the results file and when our summary script runs over the results it’ll mark the test as a known fail due to the bug referenced in the assertion message. A particularly awesome feature!
  4. Every test that hasn’t been marked as a known fail (environment or known bug) gets added to a list of tests that need to be investigated. They’ve already been run twice (see above), so there’s a very high chance that a regression bug has crept in.
  5. As well as a count of the various flavors of failed tests, we also gave percentages. These mean far more to the scrum masters, so that’s what we gave them.

Making the effort public

Initially, there was a lot of hostility toward the project – understandably! UI testing doesn’t have a good history where I work at the moment. One of the scrum masters pointed out that the best PR was fast, consistent and accurate test results and he was right. But there are a few more things that gave people reason to change their opinion.

  1. Every time our tests found a regression bug, we updated our “bug count” on a big, visible-to-all white board. There was no denying the value that we provided (we crossed the 100 bug count in only a few weeks)
  2. We gave the scrum masters remote access to our run machine. This allowed them to kick off test runs and watch them as they progressed. The instant feedback they got blew them away. The Exposé functionality let them see environment instability issues as soon as they occurred allowing them to go and kick some heads in the infrastructure team as soon as there was a problem.
  3. We put our test execution box on the end of our desk, facing out. As mentioned above, the Exposé function was great PR for the team. In the weeks before go-live, many evenings were spent huddled around the screen as we would do test run after test run with 6 or 7 people from varying levels of management all hypnotized by the tiny windows!
  4. The test owners wrote their tests on a wiki page – as soon as a test was automated we would mark it as such. Test owners knew exactly where they stood.

Summary of the lessons we learned

  1. Sell the idea of watir-based automated UI testing with working code
  2. Manage your managers’ expectations well
  3. Keep the framework simple – complexity you don’t need introduces risk you probably can’t afford
  4. Use the right tool for the right job
  5. Use free tools where possible; you’ll be freed from the hassle of purchasing stuff
  6. Use version control – it will be your salvation many times over
  7. For object-recognition, use xpath unless you have reliable IDs you can depend on
  8. Attempt to recognize environment-related errors in the test and report on them as such (watir checkers)
  9. Keep as much of your framework code as modular and independent as possible
  10. Make the test team’s work public. Provide test results as soon as is possible.
  11. Put a lot of effort into making results sanitization as quick as possible. There is very little as frustrating for a scrum master/test manager as knowing that test execution is complete but that the results “aren’t ready yet”.

That’s all for now!

Quick and easy Watir test suites with Test::Unit

Posted on by
3

A test team usually has a need for a few fixed test suites, eg: a sanity suite or a suite which contains all tests for a full run. There’s usually also a need to be able to create suites with arbitrary tests in it, eg: a suite that tests all account management functionality or a suite that runs all tests for a particular platform. There is often the need to quickly throw together a suite which can be used to regression-test a specific area of functionality.

If you’re using a ruby + watir + test::unit framework, there’s a simple way to get all of the above flexible suite buliding functionality. Here’s how…

First of all, a few prerequisites… They are:

  1. All your TestCase class files should be in one folder
  2. Your TestCase class file names should follow a strict and scalable naming convention

For the purposes of this blog post, we’ll use a simple naming convention: all test files in our imaginary project begin with test_ , all suite files begin with suite_ and all test files that test account functionality contain the word account in their file name.

Test::Unit has the concept of ‘require files’… any file that requires a TestCase file will cause the test to be executed (unless something has been done to prevent tests from being run). These require files can be used as test suites – any tests that are ‘required’ in the test suite will get executed! So, the first way you could use these require files is to individually require each TestCase file. Here’s an example… a file called suite_account_tests.rb containing the following lines; each one will load a file containing a TestCase file:

require "test_account_join"
require "test_account_close"
require "test_account_upgrade"
require "test_account_change_details"

This works, but it is laborious to build, and even more of a pain to maintain. There has to be a better way… and here it is: instead of requiring each individual file that has the word account in it, you can get ruby to do the work for you. Replace the contents of suite_account_tests.rb with the following line:

Dir.glob(File.join(File.dirname(__FILE__), '*account*.rb')).each {|f| require f }

That one line will read in all ruby files that contain the word ‘account’ in the file name, and that live in the same directory as the suite_account_tests.rb file. Then, because this is just a test::unit require file, all the TestCase classes that got required will get executed. Awesome. No need to update the suite file if an account-related TestCase file is added or deleted; it will pick up any changes automatically. All that’s required is a good naming convention…

The power of that one-liner lies in the Dir.glob function. It takes filename ‘patterns’ (shame it doesn’t take a regex) documented in the Dir.glob rdoc to decide which files to run. If you’ve got a strict naming convention, you’ll find that the Dir.glob functionality lets you create suites very quickly. If you can’t create suites quickly now, you’ll find that it’s great PR for the test team when you can!

Using these patterns, with the following one-liner we can build a suite that will run all tests:

Dir.glob(File.join(File.dirname(__FILE__), 'test_*.rb')).each {|f| require f }

All files that begin with test_ will get required and executed. This works because of the file naming convention we’re using. Only TestCase files begin with test_, so we can be sure that only tests are getting loaded. Specifically, suite files won’t be loaded as their file names begin with suite_, not test_. Anyway, you get the idea.

There’s nothing stopping you from combining the two approaches. You can have a require file that uses both the Dir.glob approach that also has individual requires if the suite needs to include specific TestCases classes.

One non-obvious advantage that this Dir.glob approach to suite files gives you is that you won’t ‘lose’ tests any more. I’ve found that when I’ve used the individual-require approach, on occasion I forget to add a require to the suite when I create a new TestClass file. I end up with tests that gather dust – they never get run. They are forgotten and left to rot. By the time I find them again, the tests are so out of date that they often need rewriting, never mind editing!

Ruby gives you a whole load of power for free; you may as well use it!