I needed to get a list of all the tags used in a cucumber project, and what follows is the formatter I wrote to list them all for me.
To use it, save the code in a file called list_tags.rb and then execute the following command:
cucumber -d -f Cucumber::Formatter::ListTags
Done!
The page object model works very well, but there are a few traps you can fall into. Alister Scott goes through a few of them and mentions one that particularly grates me: “Pages stored as instance variables”. Here’s a demonstration of the problem:
So why is that a problem? Well, there’s lots of noise – 3 lines are there just to create variables. This significantly reduces readability as you’ll end up with many, many lines of test code just creating instances of page objects. You also now have a whole load of instance variables to keep track of: @login_page, @account_page, @account_history_page. And when you’re using cucumber to run your tests, this will lead to *big* maintenance issues. I’ve had to rescue a few cucumber-based test projects and one of the most frequent causes for test-rot is that the testers lost track of their instance variables. Been in this situation before?
“Can I use @account_page here? Did I previously declare it? Hmmm… No, it’s nil when I try to use it. OK, I’ll instantiate it here. [runs the tests]. Cool, that works. Oh no! Doing that has broken some other tests that referenced @account_page but expected something else!?!? Should I fix up the other tests? Rename the @account_page variable to something else? If I do that will I break anything else?” Not fun. Big spaghetti problems.
But what to do about it? Alister suggests using blocks (provided by the page-object gem) that look something like the following:
On first glance, this looks great. No instance variables to keep track of. Just deal with the classes themselves and use only local variables inside the blocks. Nothing to keep track of. Great!
But…
The above proposal causes other maintenance hassles – the page object’s class name is now scattered throughout the code. Lots of “visit LoginPage” all over the place. What happens when the class name changes? You’ll have to make changes throughout the code. Not fun.
Solution: use an instance of an App class, this App class being a representation of the app you’re testing (the whole app, not individual pages). This App class contains one method per page class, each of these methods return an instance of the relevant page class.
“Whhhaaattt?”
OK, here’s an example:
In your tests, you would then have the following line:
When I visit the login page
…which would match the following step…
If you structure your tests such that they always begin by mentioning where the user starts (a good idea as it gives context), you can rely on the fact that @app has been instantiated so you can just use it. For example:
So how is this an improvement? Well, there’s no need to manage instance variables for different pages – just call methods on a known instantiation of the App class(@app) and they’ll return instances of the pages you want. There’s no need to mention class names; they are hidden behind methods. If the class name changes you only need to make one change (change the class referenced in the methods in the App class).
Subjective statement: I’d also argue that you also get great readability with this way of structuring things.
This is how I’ve normally organised things. And it’s worked great both on small projects of only 10′s of tests to large projects where the number of tests is 1000+. It’s the best solution I’ve come up with, it doesn’t suffer from having instance variables all over the place, neither are there class names all over the place.
I’ve written up in brief how this works if you’re using SitePrism to manage your page objects: http://rdoc.info/gems/site_prism/file/README.md#Epilogue
Sometimes cucumber’s Before hook just doesn’t cut it. Here’s a nice hack that allows you to perform some logic during execution of cucumber scenarios when a tag is first come across:
It’s a cucumber formatter that detects when a tag is first come across. When a new tag is found, the perform_logic_for method is called, passing the tag to it. Once you’re in the perform_logic_for method, you can do what you like. I’m doing stuff like checking to see if a directory matches the tag name; if I find one, I load the data into an environment (a great example of when the Before tag isn’t enough). You have free reign in here to do what you like with the tag.
To use it, save the file to features/support/tag_logic.rb, add your own tag logic, and then execute it with:
#cucumber -f pretty -f TestManagement::TagLogic -o /dev/null
I hope that helps!
Small print:
Use the Before hook instead of this if you possibly can.
After figuring out how to take a screenshot using .Net, I translated the C# I came up with to IronRuby. Now, when any of my IronRuby-powered tests fail, I take a screenshot – saves loads of time when trying to work out why a test failed!
Here’s the code you need:
require 'System.Drawing'
require 'System.Windows.Forms'
bitmap = System::Drawing::Bitmap.new(
System::Windows::Forms::Screen.PrimaryScreen.Bounds.Width,
System::Windows::Forms::Screen.PrimaryScreen.Bounds.Height,
System::Drawing::Imaging::PixelFormat.Format32bppArgb)
System::Drawing::Graphics.FromImage(bitmap).CopyFromScreen(
System::Windows::Forms::Screen.PrimaryScreen.Bounds.X,
System::Windows::Forms::Screen.PrimaryScreen.Bounds.Y,
0,
0,
System::Windows::Forms::Screen.PrimaryScreen.Bounds.Size,
System::Drawing::CopyPixelOperation.SourceCopy)
bitmap.Save("c:\\screenshot.png", System::Drawing::Imaging::ImageFormat.Png)
A screenshot is taken of the primary screen (and *only* the primary screen) and is saved as a PNG to c:\screenshot.png . Modify to your heart’s content.
Hole-in-the-open-source-market Alert: cross platform gem for taking screenshots. Please don’t make me write it!
There aren’t enough UI test automation success stories documented on the net, so here’s my contribution.
This post is about how we started with zero automated UI testing capability, and how a couple of months later we had the capability to run 1,000 tests in 20 minutes (you may also notice some gorilla-style chest-beating). Not only that, our test execution results were always spot on (no false passes, no false fails), and our framework was easy to maintain and extend. Here’s how we did it and the lessons we learned along the way.
After several false starts with UI testing (involving QTP and watin and fitnesse and and and…), it finally got off the ground after I demonstrated what could be achieved with watir. This was the first lesson for me: demonstrating working code makes for a far better case than “just believe me, I’ve done it before, it’ll work fine”. Realizing this, I spent a weekend putting together a simple ruby/test::unit/watir testing framework.
When, on monday morning, I demoed a bunch of working tests that could run over and over and over and each time give the correct results to a room full of doubters, it was decided that we’d give ruby/test::unit/watir a go.
In order to avoid the biggest mistake made in previous attempts at UI automation, I insisted on agreement to a very important constraint: the ruby/test::unit/watir framework would only test the web app part of the system. One of my philosophies when it comes to UI automation is: “use the right tool for the right job”. There would be no scope creep. The framework would be used for the web app only. The WPF fat client could be tested with something else – anything else, just please don’t subvert ruby/test::unit/watir for testing anything else but the web app.
This was all part of expectations management; over the years I’ve learned that this is one of the most important things when doing automated UI testing. It’s worth a separate post – I’ll get around to it.
The framework I came up with that weekend was very simple (and very unoriginal – I’ve done similar stuff before): each page in the app would be represented by a ruby module. Each “page module” would contain one method per control on the web page; the method when called would return the required object. Eg:
module HomePage
def home_page_sign_in_link
@browser.link(:xpath, "//a[contains(@href,'signin')]")
end
def home_join_button
@browser.button(:id, "join")
end
end
My experience is that IDs on objects often can’t be trusted. What I’ve found is that they usually get put in during development and that they are removed when they’re not required. If I write my object recognition code to depend on an ID I usually regret it. I’ve learned that xpath is often more reliable – IDs tend to disappear. Where I have reliable IDs, I’ll use them. But for anything else, I use xpath. I find myself maintaining ID-based object recognition a lot more than xpath-based object recognition. Your mileage may vary.
Each of these page modules was “mixed in” to a common module which was then mixed into the test, making each object in the system available to the test without having to deal with namespacing. Not the prettiest solution, but it got us working very quickly. We surprised ourselves by having no method name clashes – mainly due to the object naming convention we used.
Our app uses restful services so the net::http libraries were great. The calls were wrapped up in classes and placed alongside our test-data-generation code (random strings, valid usernames, etc). We wrote them to be independent of the rest of the framework (apart from the central config file that pointed them at the right environment) so that we could use them whenever and however we liked. This philosophy was applied to as much of the framework as possible – to the greatest extent possible we kept everything independent and modular. This allowed us to keep up with sweeping changes in the app we were testing. We had a lot of sweeping changes; to many people’s amazement (not least our own!) we were able to keep up with everything thrown at us.
One of watir’s greatest features which doesn’t get nearly enough air time is the “checkers” functionality. Every time a page is visited you can run custom code you’ve wrapped up in a ruby proc. Environment instability usually manifests itself in obvious ways. Watir checkers allow you to recognize these as soon as they appear. If, in your checker code, you raise an exception with a fixed message, you can find these when the test is done and mark the tests as “failed due to environment instability”; or you can be more specific, e.g: “failed due to database connection going down, again”. This saves us hours of work every day. Literally, hours.
Our test execution script keeps note of tests that failed. When all the tests have been run, the tests that failed get run again in the hope that environment instability “got better”. We’d have to rerun those tests manually at the end of a run anyway, so why not automate it? A big time saver.
We used test::unit for executing the tests. It’s simplicity and rock-solid nature made it a good choice to start the framework with. It is also easily expanded/monkey-patched – we added a few things to it.
Though I have ruby experience, everyone else on the team had a C#/NUnit background; test::unit helped bridge their knowledge gap. Conceptually, test::unit and NUnit are identical (both are xunit tools) – the similarity meant one less thing for the others to learn. When under pressure, that was a good thing.
Netbeans‘ integration with test::unit is great, so we decided on using it for our IDE. It turns out that NetBeans is a great IDE for watir testing; I heartily recommend it. The green ticks and red crosses were comforting to all the right people, the ease of viewing the source code of the watir/firewatir libraries was great, its intellisense wasn’t as bad as I was expecting it to be and its svn integration is better than anything else I’ve used so far. It’s also free so I didn’t have to get approval to buy it! It has loads of useful plugins too.
We used firefox (and therefore firewatir) because we needed to spoof useragents and headers – using firefox profiles meant we could test the mobile, browser and fat client webstores. Using firefox meant we could use plugins; here are the ones we found most useful:
One area where testing is usually decades behind development is in the use of version control. This is probably due to the fact that most commercial tools make it less than trivial to do (the QTP and RFT test file formats are hardly version-control-friendly).
One of the many joys of using watir for UI testing is that using version control is simple – there are no ‘project files’ to worry about, no accompanying files that you need to check in when you change a test file, no bizarre links across directories, no weird inter-file dependancies.
Using version control for our tests and framework meant that we were free to change stuff as we saw fit – if something didn’t work we could just revert. I can’t recommend using version control enough. The “blame” feature in svn was particularly useful 
We got our super-fast execution time by cheating: we parallelized the execution of the tests. We used a Mac Pro (16GB RAM, 8 core, 4xHDD), the Parallels virtualization software to run 12 VMs, and a shonky “n mod 12″ function to decide what tests to run on each VM (we tried to get testjour working but it needs a fair bit more dev and documentation before it works as advertised and becomes usable).
One advantage of using a Mac (apart from its rock-solid stability) was “Exposé” – we could see all 12 VMs running at once. The amount of people who walked past our desk and got hypnotized by seeing 12 tiny PCs each executing as if they were self aware… – good PR for our team!
It doesn’t matter if you can run a million tests in 5 minutes, if the results aren’t accurate and sanitized quickly. We’re quite proud of the fact that we provide our results within 5 minutes of finishing the last test (usually straight away). We’ve put a lot of effort into being able to do this and it’s one of the biggest selling points of the framework we’ve written. Here’s what we’ve done to get to this stage:
Initially, there was a lot of hostility toward the project – understandably! UI testing doesn’t have a good history where I work at the moment. One of the scrum masters pointed out that the best PR was fast, consistent and accurate test results and he was right. But there are a few more things that gave people reason to change their opinion.
That’s all for now!
I’ve recently found a couple of instances of tests with the same test name in the same test::unit TestCase class. This is a bad thing, because the when the ruby interpreter reads the file in, it will ignore all methods by the same name apart from the last one, which is the one it will use. Some code to explain:
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_me
puts "aardvark"
end
def test_me
puts "zebra"
end
end
If you run this code, you’ll get “zebra” printed to the console, not “aardvark”, because the last method definition for test_me is the one that is used.
You’ve probably worked out how this can be a very bad thing in testing – if in a TestCase class you have several test methods by the same name (copy-and-paste mistakes), you’ve got tests which don’t get executed! Then, as happened to me, you’ll get a dev coming over to you asking why your tests didn’t find a particular bug – the test which would have found the bug was never run, that’s why!
To find all instances of duplicate test names, I wrote the code below. To use it, save it to a file, set the root_test_folder variable to point to your root testing folder, and run it. If it finds duplicate tests, it’ll print out the file containing them, the duplicate test names, and how many times the test name is defined in the offending file.
root_test_folder = "/Users/nat/dev/my_testing_project/test"
unless File.exist?(root_test_folder) && File.directory?(root_test_folder)
abort("You haven't specified a directory in root_test_folder")
end
all_test_files = []
Dir.glob(File.join(root_test_folder,'**/*.rb')).each do |file|
f = File.new(file)
all_test_files << file if f.read =~ /class\s+.*\s+< Test::Unit::TestCase/
end
all_test_files.each do |test_file|
test_names = []
test_file_contents = File.new(test_file).read
test_file_contents.each_line do |line|
test_names << line.strip.gsub(/^(\s)*def\s+/,"") if line =~ /^(\s)*def\s+test_[\d\w]+(\s)*$/
end
has_duplicate_test_name = false
b = test_names.inject(Hash.new(0)) {|h,i| h[i] += 1; h }
b.each {|dupe_test_name,count| has_duplicate_test_name = true if count > 1 }
if has_duplicate_test_name
puts test_file
b.each {|dupe_test_name,count| puts "#{count}x #{dupe_test_name}" if count > 1 }
puts ""
end
end
NB:
- it assumes that each test file only contains one TestCase class
- it has very little error checking and is quite inefficient – but it does the job
- if you have any enhancements, please leave a comment
A test team usually has a need for a few fixed test suites, eg: a sanity suite or a suite which contains all tests for a full run. There’s usually also a need to be able to create suites with arbitrary tests in it, eg: a suite that tests all account management functionality or a suite that runs all tests for a particular platform. There is often the need to quickly throw together a suite which can be used to regression-test a specific area of functionality.
If you’re using a ruby + watir + test::unit framework, there’s a simple way to get all of the above flexible suite buliding functionality. Here’s how…
First of all, a few prerequisites… They are:
For the purposes of this blog post, we’ll use a simple naming convention: all test files in our imaginary project begin with test_ , all suite files begin with suite_ and all test files that test account functionality contain the word account in their file name.
Test::Unit has the concept of ‘require files’… any file that requires a TestCase file will cause the test to be executed (unless something has been done to prevent tests from being run). These require files can be used as test suites – any tests that are ‘required’ in the test suite will get executed! So, the first way you could use these require files is to individually require each TestCase file. Here’s an example… a file called suite_account_tests.rb containing the following lines; each one will load a file containing a TestCase file:
require "test_account_join"
require "test_account_close"
require "test_account_upgrade"
require "test_account_change_details"
This works, but it is laborious to build, and even more of a pain to maintain. There has to be a better way… and here it is: instead of requiring each individual file that has the word account in it, you can get ruby to do the work for you. Replace the contents of suite_account_tests.rb with the following line:
Dir.glob(File.join(File.dirname(__FILE__), '*account*.rb')).each {|f| require f }
That one line will read in all ruby files that contain the word ‘account’ in the file name, and that live in the same directory as the suite_account_tests.rb file. Then, because this is just a test::unit require file, all the TestCase classes that got required will get executed. Awesome. No need to update the suite file if an account-related TestCase file is added or deleted; it will pick up any changes automatically. All that’s required is a good naming convention…
The power of that one-liner lies in the Dir.glob function. It takes filename ‘patterns’ (shame it doesn’t take a regex) documented in the Dir.glob rdoc to decide which files to run. If you’ve got a strict naming convention, you’ll find that the Dir.glob functionality lets you create suites very quickly. If you can’t create suites quickly now, you’ll find that it’s great PR for the test team when you can!
Using these patterns, with the following one-liner we can build a suite that will run all tests:
Dir.glob(File.join(File.dirname(__FILE__), 'test_*.rb')).each {|f| require f }
All files that begin with test_ will get required and executed. This works because of the file naming convention we’re using. Only TestCase files begin with test_, so we can be sure that only tests are getting loaded. Specifically, suite files won’t be loaded as their file names begin with suite_, not test_. Anyway, you get the idea.
There’s nothing stopping you from combining the two approaches. You can have a require file that uses both the Dir.glob approach that also has individual requires if the suite needs to include specific TestCases classes.
One non-obvious advantage that this Dir.glob approach to suite files gives you is that you won’t ‘lose’ tests any more. I’ve found that when I’ve used the individual-require approach, on occasion I forget to add a require to the suite when I create a new TestClass file. I end up with tests that gather dust – they never get run. They are forgotten and left to rot. By the time I find them again, the tests are so out of date that they often need rewriting, never mind editing!
Ruby gives you a whole load of power for free; you may as well use it!
There are very few real requirements for a UI test automation framework:
Projects rarely have the time or patience to deal with “we can’t run the tests just now, we’ve got a framework issue”, so frameworks tend to get built alongside the tests; and unless you’re careful, frameworks written under these (quite common) conditions normally die in one of 2 ways:
Like most things, a middle ground needs to be found:
Hacks for hacks’ sake aren’t good. Abstractions for abstraction’s sake aren’t good either. Write what needs to be written, don’t write what doesn’t need to be written, keep things simple, and tidy up after yourself when things get hacky.