TURN (“Test::Unit Reporter (New)”) aims to fix test::unit‘s default output. Instead of waiting until the end of a test run for failure details, Turn displays failures immediately. It looks promising…
Category Archives: Making Life Easier
NetGear NV+ default admin username and password
My new NetGear NV+ has just arrived. It’s great. Apart from the fact that none of the paperwork mentions the username or password you need to log in to the admin console. If you’re looking for the same info, the details you need are:
- Username: admin
- Password: netgear1
Hope that helps!
How to test a WPF app using IronRuby and White
—UPDATE—
Since I wrote this post, I have put together a ruby gem designed for testing WPF UIs called ‘bewildr’. I wrote up an introductory post about bewildr here. Bewildr removes the need for using White – it’s written in ruby, for ruby! You’ll find its API clean and idiomatic. Anyway, back to the original post…
—————–
…and you thought ruby was only good for web testing…
So… having previously failed at getting ruby-based automated testing of WPF working, I attacked it from a different angle and succeeded! The difference is that this time I used IronRuby instead of ruby. In doing so, the White Automation library could be used out-of-the-box! It’s time to throw away your expensive, proprietary test tools and replace them with open source tools instead! Here’s how it’s done:
Prerequisites
- Install .Net Framework 2.0, 3.0 (and possibly 3.5 – I’ve got it installed and haven’t tested this stuff without it)
- Install IronRuby (but put it in c:\ironruby, not the default location that contains spaces in the path)
- Download the latest version of White (0.19 at time of writing)
- Install UISpy.exe to investigate objects on screen
- Bookmark the White API – you’ll use this a lot
I’m presuming that you know to run ruby files in IronRuby using the ir command, not the standard ruby command, iirb instead of irb and irake instead of rake.
Requires, Dlls and some helpful methods
First thing, you need to gain access to White in your project. To do that, put the following at the top of your app (eg: your env.rb file in cucumber):
$LOAD_PATH << File.expand_path("lib/White_Bin_0.19")
require "White.Core.dll"
require 'UIAutomationTypes, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
module IronRubyHelpers
def r_array_to_cs_array(args)
System::Array[System::String].new(args.map {|arg| arg.to_s.to_clr_string})
end
def r_string_to_c_string(r_string)
System::String.new(r_string)
end
end
This requires some explanation… The first line puts the directory containing the White.Core.dll file on the load path. In this case, I’ve taken the White_Bin_0.19 directory that was created when I expanded the zip I downloaded from the White site, and put it inside the lib directory in my NetBeans project. The next line requires the White library. The line after that loads the UIAutomationTypes.dll – you’ll need this if you want to find WPF objects by their type.
Next, the IronRubyHelpers module… it contains 2 methods: one converts a ruby array to a dot net array (required whenever a White object method takes an array as an argument, eg: menu interaction), and the other converts a ruby string to a dot net string (sometimes required when a White object arg takes a string).
That’s all the setup you need.
Starting and Killing WPF Apps
Now we start dealing with White… Here’s how to start a WPF app :
@app = White::Core::Application.Launch("c:\my_app\example.exe")
And here’s how to kill it:
@app.kill
The Launch(string) method is one of many ways to start an app. Check out the Application.cs documentation to see how else it can be done (I can’t get AttachOrLaunch to work – ymmv).
Grabbing Windows
You get access to objects by using the window that they’re in. That means you need to get the window first. Here’s how it’s done:
@main_window = @app.get_windows.select{|window| window.title == "My Window Title"}.first
That will get you a window whose title matches “My Window Title” exactly. If you want a bit more flexibility (eg: if your app displays its version number in the title and you don’t want to have to change your recognition string everytime you get a new build), you can use the following regex-powered window finder:
@main_window = @app.get_windows.select{|window| window.title =~ /My App, version (.*)/}.first
Your window takes a while to appear? Here’s how to wait for it:
Timeout.timeout(10) do
sleep 0.2 until @app.get_windows.select{|window| window.title == "My App"}.size > 0
end
Now you’ll wait for up to 10 seconds checking ~5 times a second to see if a window entitled “My App” appears (it only takes a tiny tweak to make it use regex). If it takes longer than 10 seconds you’ll get a Timeout exception.
Grabbing and Interacting with Objects
Now that we can get hold of windows, we can look inside them and get access to the object that they contain. White uses static methods on SearchCriteria to access object. Generally, you’ll need 2 bits of info:
- How to search (eg: by id, by control type, by text, etc)
- The value to search for (eg: “btnLogin”, ControlType.Edit, “Login…”)
Here are some examples:
@chk_visibility = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByAutomationId(r_string_to_c_string("visibleCheckBox")))
Hey, no one said it was pretty. Here we’re getting a checkbox that lives in the main window. We’re identifying it by its automation id, which happens to be “visibleCheckBox”. Another example:
@cmb_zoom = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByControlType(System::Windows::Automation::ControlType.ComboBox))
Here we’re getting the only combo box in the main window. This time we’re searching by ControlType. You can find a full list of the available control types here under the “Fields” section.
There are a few more ways to use SearchCriteria, you can read about them in the SearchCriteria.cs documentation.
And now to object interaction… Once you’ve got your object reference, you can call methods on it and hope it responds. Here are some examples:
@chk_visibility.checked
…will return true or false based on whether the checkbox is checked – read about the available methods on checkbox here. Another example:
@cmb_zoom.select(r_string_to_c_string("200%"))
This will select the “200%” option from the combo box.
Depending on the object type you could call .click, .value = "hello", .double_click, etc.
Navigating the Window Menu
Since using the window menu doesn’t fit the standard object-locator pattern, we’ll cover that here.
def select_menu_item(*args)
my_menu_bar = @main_window.menu_bar
my_menu_item = my_menu_bar.menu_item(r_array_to_cs_array(args))
my_menu_item.click
end
select_menu_item("File", "Open...")
Using window menus is a bit weird in White. The easiest way to go about it is to use the select_menu_item above. It gets the menu bar out of the window, does some ruby-to-dot-net magic with the arguments and then clicks the menu item. I’ve included an example call to the method that does the standard File->Open action.
A few more things
Because we’re using IronRuby, there’s a quirk you need to be aware of – you may have spotted it already… IronRuby translates White’s CamelCase methods to the standard ruby_underscore_case. Eg:, if the White documentation says that a particular object supports the “DoubleClick” method, you can call that method in your ruby scripts with “double_click”. You get used to it.
Speed and Reliability
…or lack of same. White is slow. And buggy. IronRuby startup time is slow too. I’ve found loads of places where White claims to do something (eg: table interaction), but what’s there isn’t unusable. Some of the stuff that does work only works often, not all the time. I’ve rewritten some of the buggy functionality using the low level interaction libraries provided by microsoft and the replacement code runs several orders of magnitude faster than white. Please don’e expect fast execution – you just won’t get it. Be prepared for molasses-speed testing.
Example project using Paint.NET
I’ve put together a badly designed set of tests around Paint.NET (3.5.3 at time of writing) using Test::Unit. It’s a NetBeans project, but even so you’ll need to run the tests off the command line (there’s no IronRuby-NetBeans support that I can find). Make sure to have Paint.NET installed in its default location and then try running “irake test” in the project directory… and have patience – it crawls. It takes around 2 minutes to run 4 tests.
>>> Download my example project here <<<
Enjoy!
Where on earth is UISpy?
The very useful and very frustrating UISpy.exe seems to have gone missing from the latest Windows SDKs. After a bit of hunting around (and several gig of downloads later…) I found it in the “Windows SDK for Vista Update” (works for XP too…); the link is:
If all you want is UISpy, just select ‘Windows Development Tools’ option in the installation options window – you don’t need the rest.
Updated script to list all cucumber step definitions
In a previous post I put up a small script that would dump out all step definitions available in a cucumber project. Here’s an updated version… it hasn’t changed much apart from the output is now a html table that for each step definition contains the regular expression, any regex modifiers, any parameters to the step definition and a link to the file that contains the definition.
Anyway, here’s the script:
step_definition_dir = "./features/steps"
f = File.new("output.htm", "w")
f << "<table><th>Regex</th><th>Modifiers</th><th>Step Definition Args</th><th>Source file</th>"
Dir.glob(File.join(step_definition_dir,'**/*.rb')).each do |step_file|
File.new(step_file).read.each_line do |line|
next unless line =~ /^\s*(?:Given|When|Then)\s+\//
matches = /(?:Given|When|Then)\s*\/(.*)\/([imxo]*)\s*do\s*(?:$|\|(.*)\|)/.match(line).captures
matches << step_file
f << "<tr>"
f << "<td>#{matches[0]}</td>"
f << "<td>#{matches[1]}</td>"
f << "<td>#{matches[2]}</td>"
f << "<td><a href=\"#{matches[3]}\">#{matches[3]}</a></td>"
f << "</tr>"
end
end
f << "</table>"
—— UPDATE ——
Turns out that in the latest version of cucumber (0.6.1) there’s a new formatter called stepdefs which prints out the step regexs and the step definition file that they live in. It doesn’t report regex modifiers or the step arguments – apart from that it’s great. Here’s how to use it:
cucumber -d -f stepdefs
…though it only seems to work for steps called in an ordinary scenario, not a scenario outline.
Find duplicate test names in ruby test::unit files
I’ve recently found a couple of instances of tests with the same test name in the same test::unit TestCase class. This is a bad thing, because the when the ruby interpreter reads the file in, it will ignore all methods by the same name apart from the last one, which is the one it will use. Some code to explain:
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_me
puts "aardvark"
end
def test_me
puts "zebra"
end
end
If you run this code, you’ll get “zebra” printed to the console, not “aardvark”, because the last method definition for test_me is the one that is used.
You’ve probably worked out how this can be a very bad thing in testing – if in a TestCase class you have several test methods by the same name (copy-and-paste mistakes), you’ve got tests which don’t get executed! Then, as happened to me, you’ll get a dev coming over to you asking why your tests didn’t find a particular bug – the test which would have found the bug was never run, that’s why!
To find all instances of duplicate test names, I wrote the code below. To use it, save it to a file, set the root_test_folder variable to point to your root testing folder, and run it. If it finds duplicate tests, it’ll print out the file containing them, the duplicate test names, and how many times the test name is defined in the offending file.
root_test_folder = "/Users/nat/dev/my_testing_project/test"
unless File.exist?(root_test_folder) && File.directory?(root_test_folder)
abort("You haven't specified a directory in root_test_folder")
end
all_test_files = []
Dir.glob(File.join(root_test_folder,'**/*.rb')).each do |file|
f = File.new(file)
all_test_files << file if f.read =~ /class\s+.*\s+< Test::Unit::TestCase/
end
all_test_files.each do |test_file|
test_names = []
test_file_contents = File.new(test_file).read
test_file_contents.each_line do |line|
test_names << line.strip.gsub(/^(\s)*def\s+/,"") if line =~ /^(\s)*def\s+test_[\d\w]+(\s)*$/
end
has_duplicate_test_name = false
b = test_names.inject(Hash.new(0)) {|h,i| h[i] += 1; h }
b.each {|dupe_test_name,count| has_duplicate_test_name = true if count > 1 }
if has_duplicate_test_name
puts test_file
b.each {|dupe_test_name,count| puts "#{count}x #{dupe_test_name}" if count > 1 }
puts ""
end
end
NB:
- it assumes that each test file only contains one TestCase class
- it has very little error checking and is quite inefficient – but it does the job
- if you have any enhancements, please leave a comment
Quiet windows command line – no output please!
I often need to call a command line app but I don’t want it to output anything. I just want it to do its job and then die quietly. Here’s how to silence an app… stick the following on the end of the command:
>nul 2>&1
Done.
Quick and easy Watir test suites with Test::Unit
A test team usually has a need for a few fixed test suites, eg: a sanity suite or a suite which contains all tests for a full run. There’s usually also a need to be able to create suites with arbitrary tests in it, eg: a suite that tests all account management functionality or a suite that runs all tests for a particular platform. There is often the need to quickly throw together a suite which can be used to regression-test a specific area of functionality.
If you’re using a ruby + watir + test::unit framework, there’s a simple way to get all of the above flexible suite buliding functionality. Here’s how…
First of all, a few prerequisites… They are:
- All your TestCase class files should be in one folder
- Your TestCase class file names should follow a strict and scalable naming convention
For the purposes of this blog post, we’ll use a simple naming convention: all test files in our imaginary project begin with test_ , all suite files begin with suite_ and all test files that test account functionality contain the word account in their file name.
Test::Unit has the concept of ‘require files’… any file that requires a TestCase file will cause the test to be executed (unless something has been done to prevent tests from being run). These require files can be used as test suites – any tests that are ‘required’ in the test suite will get executed! So, the first way you could use these require files is to individually require each TestCase file. Here’s an example… a file called suite_account_tests.rb containing the following lines; each one will load a file containing a TestCase file:
require "test_account_join"
require "test_account_close"
require "test_account_upgrade"
require "test_account_change_details"
This works, but it is laborious to build, and even more of a pain to maintain. There has to be a better way… and here it is: instead of requiring each individual file that has the word account in it, you can get ruby to do the work for you. Replace the contents of suite_account_tests.rb with the following line:
Dir.glob(File.join(File.dirname(__FILE__), '*account*.rb')).each {|f| require f }
That one line will read in all ruby files that contain the word ‘account’ in the file name, and that live in the same directory as the suite_account_tests.rb file. Then, because this is just a test::unit require file, all the TestCase classes that got required will get executed. Awesome. No need to update the suite file if an account-related TestCase file is added or deleted; it will pick up any changes automatically. All that’s required is a good naming convention…
The power of that one-liner lies in the Dir.glob function. It takes filename ‘patterns’ (shame it doesn’t take a regex) documented in the Dir.glob rdoc to decide which files to run. If you’ve got a strict naming convention, you’ll find that the Dir.glob functionality lets you create suites very quickly. If you can’t create suites quickly now, you’ll find that it’s great PR for the test team when you can!
Using these patterns, with the following one-liner we can build a suite that will run all tests:
Dir.glob(File.join(File.dirname(__FILE__), 'test_*.rb')).each {|f| require f }
All files that begin with test_ will get required and executed. This works because of the file naming convention we’re using. Only TestCase files begin with test_, so we can be sure that only tests are getting loaded. Specifically, suite files won’t be loaded as their file names begin with suite_, not test_. Anyway, you get the idea.
There’s nothing stopping you from combining the two approaches. You can have a require file that uses both the Dir.glob approach that also has individual requires if the suite needs to include specific TestCases classes.
One non-obvious advantage that this Dir.glob approach to suite files gives you is that you won’t ‘lose’ tests any more. I’ve found that when I’ve used the individual-require approach, on occasion I forget to add a require to the suite when I create a new TestClass file. I end up with tests that gather dust – they never get run. They are forgotten and left to rot. By the time I find them again, the tests are so out of date that they often need rewriting, never mind editing!
Ruby gives you a whole load of power for free; you may as well use it!
