Tag Archives: Test Case Management

Tag based logic in Cucumber

Posted on by
Reply

Sometimes cucumber’s Before hook just doesn’t cut it. Here’s a nice hack that allows you to perform some logic during execution of cucumber scenarios when a tag is first come across:

It’s a cucumber formatter that detects when a tag is first come across. When a new tag is found, the perform_logic_for method is called, passing the tag to it. Once you’re in the perform_logic_for method, you can do what you like. I’m doing stuff like checking to see if a directory matches the tag name; if I find one, I load the data into an environment (a great example of when the Before tag isn’t enough). You have free reign in here to do what you like with the tag.

To use it, save the file to features/support/tag_logic.rb, add your own tag logic, and then execute it with:

#cucumber -f pretty -f TestManagement::TagLogic -o /dev/null

I hope that helps!

Small print:
Use the Before hook instead of this if you possibly can.

Is Cucumber adding value to your project? The Chicken Test!

Posted on by
2

Don’t get me wrong – I love cucumber. I’ve been on projects where it has been used successfully and I’d partly attribute the success of those projects to the use of cucumber. It is a fantastic tool for BDD. When used right:

  • its scenarios provide a definitive specification of app functionality
  • its output provides one of the most useful metrics of progress: running passing tests
  • it gets everyone talking the same language
  • it provides a layer of abstraction between the required behavior and the app implementation (if you keep your steps declarative)
  • it helps to keep the focus on what code really needs to be written

All of this from one tool! Pretty good, huh? Well, like anything good, it comes with costs – those’ll be covered in a later post.

Whether the costs are outweighed by the benefits that cucumber can provide is dependent on how closely the customers, devs and testers are working together. The more blurred the roles of dev and test are and the more involved the customer is, the more value you’ll get from using cucumber. The further the devs are from the testers and the less involved the customer is, the less value cucumber will give you.

There comes a point where the costs of using cucumber outweigh the benefits – and that point comes along pretty quick IMHO. Most of the projects that I’ve seen using cucumber really shouldn’t be. Yes, it’s the flavor of the month. Yes, it’s what all the cool kidz are using. But it’s not a panacea. Using cucumber will not shower your project with pink unicorns or make your team collaborative. Too often I’ve seen cucumber used for CVDD/RDD instead of BDD – almost always a bad move…

Having been on quite a few projects that have used cucumber, I’ve come up with an unscientific and clumsy acid test that will show you if you should stop using cucumber and move to something with lower maintenance costs (eg: rspec/testunit/whatever). Here, in all its glory, is The Chicken Test™:

Instead of your usual feature description, try putting the following at the top of your next feature file:

Feature: [whatever feature is being written]
  As a chicken
  I want to cross the road
  In order to get to the other side

Possible results of The Chicken Test:

  • You get caught while typing - Pass. To have got caught this early you must be writing your scenarios collaboratively. Continue using cucumber.
  • You manage to get it checked in - I have a bad feeling about this… Get people talking again.
  • No one notices for a couple of days even though the scenarios are being run - Fail. If people aren’t even reading the scenarios, they’re not serving their purpose. Use rspec.
  • No one notices for a couple of months – Epic fail. Wipe that smug look off your face – you’ve made your point. The devs don’t want you testers “in the way”, so you may as well make life easier for yourself – move to rspec.

All too often, projects I’ve seen fail the chicken test. Does yours?

Programmatically take screenshot in IronRuby

Posted on by
2

After figuring out how to take a screenshot using .Net, I translated the C# I came up with to IronRuby. Now, when any of my IronRuby-powered tests fail, I take a screenshot – saves loads of time when trying to work out why a test failed!

Here’s the code you need:


require 'System.Drawing'
require 'System.Windows.Forms'

bitmap = System::Drawing::Bitmap.new(
        System::Windows::Forms::Screen.PrimaryScreen.Bounds.Width,
        System::Windows::Forms::Screen.PrimaryScreen.Bounds.Height,
        System::Drawing::Imaging::PixelFormat.Format32bppArgb)

System::Drawing::Graphics.FromImage(bitmap).CopyFromScreen(
        System::Windows::Forms::Screen.PrimaryScreen.Bounds.X,
        System::Windows::Forms::Screen.PrimaryScreen.Bounds.Y,
        0,
        0,
        System::Windows::Forms::Screen.PrimaryScreen.Bounds.Size,
        System::Drawing::CopyPixelOperation.SourceCopy)

bitmap.Save("c:\\screenshot.png", System::Drawing::Imaging::ImageFormat.Png)

A screenshot is taken of the primary screen (and *only* the primary screen) and is saved as a PNG to c:\screenshot.png . Modify to your heart’s content.

Hole-in-the-open-source-market Alert: cross platform gem for taking screenshots. Please don’t make me write it!

Find duplicate test names in ruby test::unit files

Posted on by
Reply

I’ve recently found a couple of instances of tests with the same test name in the same test::unit TestCase class. This is a bad thing, because the when the ruby interpreter reads the file in, it will ignore all methods by the same name apart from the last one, which is the one it will use. Some code to explain:

require 'test/unit'
class MyTest < Test::Unit::TestCase
  def test_me
    puts "aardvark"
  end
  def test_me
    puts "zebra"
  end
end

If you run this code, you’ll get “zebra” printed to the console, not “aardvark”, because the last method definition for test_me is the one that is used.

You’ve probably worked out how this can be a very bad thing in testing – if in a TestCase class you have several test methods by the same name (copy-and-paste mistakes), you’ve got tests which don’t get executed! Then, as happened to me, you’ll get a dev coming over to you asking why your tests didn’t find a particular bug – the test which would have found the bug was never run, that’s why!

To find all instances of duplicate test names, I wrote the code below. To use it, save it to a file, set the root_test_folder variable to point to your root testing folder, and run it. If it finds duplicate tests, it’ll print out the file containing them, the duplicate test names, and how many times the test name is defined in the offending file.


root_test_folder = "/Users/nat/dev/my_testing_project/test"

unless File.exist?(root_test_folder) && File.directory?(root_test_folder)
  abort("You haven't specified a directory in root_test_folder")
end

all_test_files = []

Dir.glob(File.join(root_test_folder,'**/*.rb')).each do |file|
  f = File.new(file)
  all_test_files << file if f.read =~ /class\s+.*\s+< Test::Unit::TestCase/
end

all_test_files.each do |test_file|
  test_names = []
  test_file_contents = File.new(test_file).read

  test_file_contents.each_line do |line|
    test_names << line.strip.gsub(/^(\s)*def\s+/,"") if line =~ /^(\s)*def\s+test_[\d\w]+(\s)*$/
  end

  has_duplicate_test_name = false

  b = test_names.inject(Hash.new(0)) {|h,i| h[i] += 1; h }
  b.each {|dupe_test_name,count| has_duplicate_test_name = true if count > 1 }

  if has_duplicate_test_name
    puts test_file
    b.each {|dupe_test_name,count| puts "#{count}x #{dupe_test_name}" if count > 1 }
    puts ""
  end
end

NB:
- it assumes that each test file only contains one TestCase class
- it has very little error checking and is quite inefficient – but it does the job
- if you have any enhancements, please leave a comment

Quick and easy Watir test suites with Test::Unit

Posted on by
2

A test team usually has a need for a few fixed test suites, eg: a sanity suite or a suite which contains all tests for a full run. There’s usually also a need to be able to create suites with arbitrary tests in it, eg: a suite that tests all account management functionality or a suite that runs all tests for a particular platform. There is often the need to quickly throw together a suite which can be used to regression-test a specific area of functionality.

If you’re using a ruby + watir + test::unit framework, there’s a simple way to get all of the above flexible suite buliding functionality. Here’s how…

First of all, a few prerequisites… They are:

  1. All your TestCase class files should be in one folder
  2. Your TestCase class file names should follow a strict and scalable naming convention

For the purposes of this blog post, we’ll use a simple naming convention: all test files in our imaginary project begin with test_ , all suite files begin with suite_ and all test files that test account functionality contain the word account in their file name.

Test::Unit has the concept of ‘require files’… any file that requires a TestCase file will cause the test to be executed (unless something has been done to prevent tests from being run). These require files can be used as test suites – any tests that are ‘required’ in the test suite will get executed! So, the first way you could use these require files is to individually require each TestCase file. Here’s an example… a file called suite_account_tests.rb containing the following lines; each one will load a file containing a TestCase file:

require "test_account_join"
require "test_account_close"
require "test_account_upgrade"
require "test_account_change_details"

This works, but it is laborious to build, and even more of a pain to maintain. There has to be a better way… and here it is: instead of requiring each individual file that has the word account in it, you can get ruby to do the work for you. Replace the contents of suite_account_tests.rb with the following line:

Dir.glob(File.join(File.dirname(__FILE__), '*account*.rb')).each {|f| require f }

That one line will read in all ruby files that contain the word ‘account’ in the file name, and that live in the same directory as the suite_account_tests.rb file. Then, because this is just a test::unit require file, all the TestCase classes that got required will get executed. Awesome. No need to update the suite file if an account-related TestCase file is added or deleted; it will pick up any changes automatically. All that’s required is a good naming convention…

The power of that one-liner lies in the Dir.glob function. It takes filename ‘patterns’ (shame it doesn’t take a regex) documented in the Dir.glob rdoc to decide which files to run. If you’ve got a strict naming convention, you’ll find that the Dir.glob functionality lets you create suites very quickly. If you can’t create suites quickly now, you’ll find that it’s great PR for the test team when you can!

Using these patterns, with the following one-liner we can build a suite that will run all tests:

Dir.glob(File.join(File.dirname(__FILE__), 'test_*.rb')).each {|f| require f }

All files that begin with test_ will get required and executed. This works because of the file naming convention we’re using. Only TestCase files begin with test_, so we can be sure that only tests are getting loaded. Specifically, suite files won’t be loaded as their file names begin with suite_, not test_. Anyway, you get the idea.

There’s nothing stopping you from combining the two approaches. You can have a require file that uses both the Dir.glob approach that also has individual requires if the suite needs to include specific TestCases classes.

One non-obvious advantage that this Dir.glob approach to suite files gives you is that you won’t ‘lose’ tests any more. I’ve found that when I’ve used the individual-require approach, on occasion I forget to add a require to the suite when I create a new TestClass file. I end up with tests that gather dust – they never get run. They are forgotten and left to rot. By the time I find them again, the tests are so out of date that they often need rewriting, never mind editing!

Ruby gives you a whole load of power for free; you may as well use it!

How to print test names when executing Ruby Test::Unit tests

Posted on by
Reply

When I run my test_all.rb from the command line, I get ‘.’ when a test passes, but I don’t know which test it was that passed. I only get the names of tests that fail or error. Quite sensible – for regression testing, this is all you need. But where audit trails are required, I need to be able to provide a list of all tests that were executed in a particular test run. Here’s how to do it (using tests in a NetBeans project as an example)…

Instead of running:

ruby -I test -I lib test/test_all.rb

…run:

ruby -I test -I lib test/test_all.rb -v

The ‘verbose’ argument does exactly what I need! Here’s the kind of output you get:

test_that_passes(RelevantTestCaseClass): .
test_that_fails(RelevantTestCaseClass): F
test_that_errors(RelevantTestCaseClass): E

You’ll still get all the errors listed at the end of the run as usual.

Test Case Interdependency

Posted on by
1

One of the most common ways of structuring a series of test cases is to make one test case dependent on the outcome of another. For example, Test Case ‘A’ verifies the functionality surrounding the ability to create an account. Test Case ‘B’ verifies functionality surrounding account deletion, but instead of stating that the required data is and account in a particular state, it states that the account generated by test case ‘A’ should be the one to test for deletion. The mistake cascades through the test cycle: in execution of the test suite, if test case ‘A’ fails then test case ‘B’ cannot be executed and so it is marked as ‘failed’.

This test case interdependency causes problems for automation. It’s also a bad thing to do in general. Why?

In the above example, when it comes down to it, test case ‘B’ is not dependent on test case ‘A’ at all. If ‘B’ is testing deletion, it should test deletion. Deletion is dependent on an account, not necessarily a specific one (i.e. the one generated by test case ‘A’). OK, the account to test deletion against may need to be in a specific state (e.g. not already marked for deletion, etc…) but that’s not the same as dictating a specific account number.

As well as being, er, “philosophically” wrong, interdependency of test cases leads to testers incorrectly failing tests. Marking test case ‘B’ as failed just because test case ‘A’ did produces incorrect data in the test report. Why? Marking a test as failed when it hasn’t been executed is wrong, no matter what the reason is. The tester executing test case ‘B’ should have picked one of the (possibly) large number of valid accounts to use instead of being limited to test case ‘A’s account. That way, the ‘delete’ functionality can be tested even if the ‘create’ functionality is broken.

How is this a problem for automation? Well, an automated test should be just that: an automatic version of a manual test. Hard-wiring data into automated tests is common (and sold as a ‘feature’ of many packages), but makes the tests very fragile. If the data doesn’t exist (due to other tests failing), some tests won’t be able to run even though there may be plenty of valid data to use!

An easy fix is to make a slight modification to your tests: change them to be dependent on data in a particular state rather than specific data. Subtle difference with a large impact on test case management and execution. You’ll still be testing the same functionality, but the tests are much less interdependent. You’ll be able to execute all your tests (not just a subset) and your automated tests will be much more reliable and maintainable.