Ruby in NetBeans is no more…

First, Apple said they’re no longer supporting Java on the Mac platform, and I feared for NetBeans’ life… “Oh no! The carpet is being pulled from under my favourite Ruby IDE’s feet! How will it survive?”. Well, it turns out that Oracle answered that question for me. Due to “limited engineering resources” (Oracle’s only a small company – worth a mere US$161bln, a trifling sum) Ruby is no longer supported. NetBeans Ruby edition is no more as of NetBeans 7.

There has been a lot of angry talk bemoaning some of the actions taken by Oracle after they bought Sun. I generally stay clear of heated debate – I’ve got work to do :P – but I’m taking the cold-blooded murder of Ruby in Netbeans personally. Oi! Larry! Just who do you think you are!? I *like* coding Ruby in NetBeans and now I can’t any more! Try not to screw up anything else please!

I can put it off no longer, I’m going to have to learn vim. Wish me luck.

Netbeans ‘Cucumber Features’ plugin in Beta!

The best ruby IDE, Netbeans, now has a Cucumber plugin in beta! It’s the old “QuBiT” plugin that I’ve been using for a while though it looks like it’s been rebranded as the “Cucumber Features” plugin. Here it is:

As well as syntax highlighting and pretty icons, the new version has the following new (to me) features:

  • right-click “Run Feature” in the editor window and the file browser window
  • Better formatting (plugin now in line with latest cucumber changes)
  • Formatting of Examples tables (killer feature for me!)

Nice! Now, if only it would provide right-click-run-scenario…

How to test a WPF app using IronRuby and White

UPDATE
Since I wrote this post, I have put together a ruby gem designed for testing WPF UIs called ‘bewildr’. I wrote up an introductory post about bewildr here. Bewildr removes the need for using White – it’s written in ruby, for ruby! You’ll find its API clean and idiomatic. Anyway, back to the original post…
—————–

…and you thought ruby was only good for web testing…

So… having previously failed at getting ruby-based automated testing of WPF working, I attacked it from a different angle and succeeded! The difference is that this time I used IronRuby instead of ruby. In doing so, the White Automation library could be used out-of-the-box! It’s time to throw away your expensive, proprietary test tools and replace them with open source tools instead! Here’s how it’s done:

Prerequisites

  • Install .Net Framework 2.0, 3.0 (and possibly 3.5 – I’ve got it installed and haven’t tested this stuff without it)
  • Install IronRuby (but put it in c:\ironruby, not the default location that contains spaces in the path)
  • Download the latest version of White (0.19 at time of writing)
  • Install UISpy.exe to investigate objects on screen
  • Bookmark the White API – you’ll use this a lot

I’m presuming that you know to run ruby files in IronRuby using the ir command, not the standard ruby command, iirb instead of irb and irake instead of rake.

Requires, Dlls and some helpful methods

First thing, you need to gain access to White in your project. To do that, put the following at the top of your app (eg: your env.rb file in cucumber):


$LOAD_PATH << File.expand_path("lib/White_Bin_0.19")
require "White.Core.dll"
require 'UIAutomationTypes, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

module IronRubyHelpers
 def r_array_to_cs_array(args)
   System::Array[System::String].new(args.map {|arg| arg.to_s.to_clr_string})
 end

 def r_string_to_c_string(r_string)
   System::String.new(r_string)
 end
end

This requires some explanation… The first line puts the directory containing the White.Core.dll file on the load path. In this case, I’ve taken the White_Bin_0.19 directory that was created when I expanded the zip I downloaded from the White site, and put it inside the lib directory in my NetBeans project. The next line requires the White library. The line after that loads the UIAutomationTypes.dll – you’ll need this if you want to find WPF objects by their type.

Next, the IronRubyHelpers module… it contains 2 methods: one converts a ruby array to a dot net array (required whenever a White object method takes an array as an argument, eg: menu interaction), and the other converts a ruby string to a dot net string (sometimes required when a White object arg takes a string).

That’s all the setup you need.

Starting and Killing WPF Apps

Now we start dealing with White… Here’s how to start a WPF app :

@app = White::Core::Application.Launch("c:\my_app\example.exe")

And here’s how to kill it:

@app.kill

The Launch(string) method is one of many ways to start an app. Check out the Application.cs documentation to see how else it can be done (I can’t get AttachOrLaunch to work – ymmv).

Grabbing Windows

You get access to objects by using the window that they’re in. That means you need to get the window first. Here’s how it’s done:

@main_window = @app.get_windows.select{|window| window.title == "My Window Title"}.first

That will get you a window whose title matches “My Window Title” exactly. If you want a bit more flexibility (eg: if your app displays its version number in the title and you don’t want to have to change your recognition string everytime you get a new build), you can use the following regex-powered window finder:

@main_window = @app.get_windows.select{|window| window.title =~ /My App, version (.*)/}.first

Your window takes a while to appear? Here’s how to wait for it:

Timeout.timeout(10) do
  sleep 0.2 until @app.get_windows.select{|window| window.title == "My App"}.size > 0
end

Now you’ll wait for up to 10 seconds checking ~5 times a second to see if a window entitled “My App” appears (it only takes a tiny tweak to make it use regex). If it takes longer than 10 seconds you’ll get a Timeout exception.

Grabbing and Interacting with Objects

Now that we can get hold of windows, we can look inside them and get access to the object that they contain. White uses static methods on SearchCriteria to access object. Generally, you’ll need 2 bits of info:

  1. How to search (eg: by id, by control type, by text, etc)
  2. The value to search for (eg: “btnLogin”, ControlType.Edit, “Login…”)

Here are some examples:

@chk_visibility = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByAutomationId(r_string_to_c_string("visibleCheckBox")))

Hey, no one said it was pretty. Here we’re getting a checkbox that lives in the main window. We’re identifying it by its automation id, which happens to be “visibleCheckBox”. Another example:

@cmb_zoom = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByControlType(System::Windows::Automation::ControlType.ComboBox))

Here we’re getting the only combo box in the main window. This time we’re searching by ControlType. You can find a full list of the available control types here under the “Fields” section.

There are a few more ways to use SearchCriteria, you can read about them in the SearchCriteria.cs documentation.

And now to object interaction… Once you’ve got your object reference, you can call methods on it and hope it responds. Here are some examples:

@chk_visibility.checked

…will return true or false based on whether the checkbox is checked – read about the available methods on checkbox here. Another example:

@cmb_zoom.select(r_string_to_c_string("200%"))

This will select the “200%” option from the combo box.

Depending on the object type you could call .click, .value = "hello", .double_click, etc.

Navigating the Window Menu

Since using the window menu doesn’t fit the standard object-locator pattern, we’ll cover that here.

def select_menu_item(*args)
 my_menu_bar = @main_window.menu_bar
 my_menu_item = my_menu_bar.menu_item(r_array_to_cs_array(args))
 my_menu_item.click
end

select_menu_item("File", "Open...")

Using window menus is a bit weird in White. The easiest way to go about it is to use the select_menu_item above. It gets the menu bar out of the window, does some ruby-to-dot-net magic with the arguments and then clicks the menu item. I’ve included an example call to the method that does the standard File->Open action.

A few more things

Because we’re using IronRuby, there’s a quirk you need to be aware of – you may have spotted it already… IronRuby translates White’s CamelCase methods to the standard ruby_underscore_case. Eg:, if the White documentation says that a particular object supports the “DoubleClick” method, you can call that method in your ruby scripts with “double_click”. You get used to it.

Speed and Reliability

…or lack of same. White is slow. And buggy. IronRuby startup time is slow too. I’ve found loads of places where White claims to do something (eg: table interaction), but what’s there isn’t unusable. Some of the stuff that does work only works often, not all the time. I’ve rewritten some of the buggy functionality using the low level interaction libraries provided by microsoft and the replacement code runs several orders of magnitude faster than white. Please don’e expect fast execution – you just won’t get it. Be prepared for molasses-speed testing.

Example project using Paint.NET

I’ve put together a badly designed set of tests around Paint.NET (3.5.3 at time of writing) using Test::Unit. It’s a NetBeans project, but even so you’ll need to run the tests off the command line (there’s no IronRuby-NetBeans support that I can find). Make sure to have Paint.NET installed in its default location and then try running “irake test” in the project directory… and have patience – it crawls. It takes around 2 minutes to run 4 tests.

>>> Download my example project here <<<

Enjoy!

Lessons from a watir success story

There aren’t enough UI test automation success stories documented on the net, so here’s my contribution.

This post is about  how we started with zero automated UI testing capability, and how a couple of months later we had the capability to run 1,000 tests in 20 minutes (you may also notice some gorilla-style chest-beating). Not only that, our test execution results were always spot on (no false passes, no false fails), and our framework was easy to maintain and extend. Here’s how we did it and the lessons we learned along the way.

The Phoenix rises from the ashes…

After several false starts with UI testing (involving QTP and watin and fitnesse and and and…), it finally got off the ground after I demonstrated what could be achieved with watir. This was the first lesson for me: demonstrating working code makes for a far better case than “just believe me, I’ve done it before, it’ll work fine”. Realizing this, I spent a weekend putting together a simple ruby/test::unit/watir testing framework.

When, on monday morning, I demoed a bunch of working tests that could run over and over and over and each time give the correct results to a room full of doubters, it was decided that we’d give ruby/test::unit/watir a go.

An important constraint

In order to avoid the biggest mistake made in previous attempts at UI automation, I insisted on agreement to a very important constraint: the ruby/test::unit/watir framework would only test the web app part of the system. One of my philosophies when it comes to UI automation is: “use the right tool for the right job”. There would be no scope creep. The framework would be used for the web app only. The WPF fat client could be tested with something else – anything else, just please don’t subvert ruby/test::unit/watir for testing anything else but the web app.

This was all part of expectations management; over the years I’ve learned that this is one of the most important things when doing automated UI testing. It’s worth a separate post – I’ll get around to it.

Framework overview

The framework I came up with that weekend was very simple (and very unoriginal – I’ve done similar stuff before): each page in the app would be represented by a ruby module. Each “page module” would contain one method per control on the web page; the method when called would return the required object. Eg:

module HomePage
  def home_page_sign_in_link
    @browser.link(:xpath, "//a[contains(@href,'signin')]")
  end
  def home_join_button
    @browser.button(:id, "join")
  end
end

My experience is that IDs on objects often can’t be trusted. What I’ve found is that they usually get put in during development and that they are removed when they’re not required. If I write my object recognition code to depend on an ID I usually regret it. I’ve learned that xpath is often more reliable – IDs tend to disappear. Where I have reliable IDs, I’ll use them. But for anything else, I use xpath. I find myself maintaining ID-based object recognition a lot more than xpath-based object recognition. Your mileage may vary.

Each of these page modules was “mixed in” to a common module which was then mixed into the test, making each object in the system available to the test without having to deal with namespacing. Not the prettiest solution, but it got us working very quickly. We surprised ourselves by having no method name clashes – mainly due to the object naming convention we used.

Our app uses restful services so the net::http libraries were great. The calls were wrapped up in classes and placed alongside our test-data-generation code (random strings, valid usernames, etc). We wrote them to be independent of the rest of the framework (apart from the central config file that pointed them at the right environment) so that we could use them whenever and however we liked. This philosophy was applied to as much of the framework as possible – to the greatest extent possible we kept everything independent and modular. This allowed us to keep up with sweeping changes in the app we were testing. We had a lot of sweeping changes; to many people’s amazement (not least our own!) we were able to keep up with everything thrown at us.

One of watir’s greatest features which doesn’t get nearly enough air time is the “checkers” functionality. Every time a page is visited you can run custom code you’ve wrapped up in a ruby proc. Environment instability usually manifests itself in obvious ways. Watir checkers allow you to recognize these as soon as they appear. If, in your checker code, you raise an exception with a fixed message, you can find these when the test is done and mark the tests as “failed due to environment instability”; or you can be more specific, e.g: “failed due to database connection going down, again”. This saves us hours of work every day. Literally, hours.

Our test execution script keeps note of tests that failed. When all the tests have been run, the tests that failed get run again in the hope that environment instability “got better”. We’d have to rerun those tests manually at the end of a run anyway, so why not automate it? A big time saver.

Test::Unit

We used test::unit for executing the tests. It’s simplicity and rock-solid nature made it a good choice to start the framework with. It is also easily expanded/monkey-patched – we added a few things to it.

Though I have ruby experience, everyone else on the team had a C#/NUnit background; test::unit helped bridge their knowledge gap. Conceptually, test::unit and NUnit are identical (both are xunit tools) – the similarity meant one less thing for the others to learn. When under pressure, that was a good thing.

Netbeans as an IDE

Netbeansintegration with test::unit is great, so we decided on using it for our IDE. It turns out that NetBeans is a great IDE for watir testing; I heartily recommend it. The green ticks and red crosses were comforting to all the right people, the ease of viewing the source code of the watir/firewatir libraries was great, its intellisense wasn’t as bad as I was expecting it to be and its svn integration is better than anything else I’ve used so far. It’s also free so I didn’t have to get approval to buy it! It has loads of useful plugins too.

Firefox plugins

We used firefox (and therefore firewatir) because we needed to spoof useragents and headers – using firefox profiles meant we could test the mobile, browser and fat client webstores. Using firefox meant we could use plugins; here are the ones we found most useful:

  • Firebug – it is awesome. Install it, learn it, love it.
  • XPath Checker – It’s a great tool which does one thing very well: Write an xpath and it’ll show you what it evaluates to. There are other tools that do the same thing and more but the experience of everyone on the team was that xpath viewer was all they needed. Everything else was bloat.

Version Control

One area where testing is usually decades behind development is in the use of version control. This is probably due to the fact that most commercial tools make it less than trivial to do (the QTP and RFT test file formats are hardly version-control-friendly).

One of the many joys of using watir for UI testing is that using version control is simple – there are no ‘project files’ to worry about, no accompanying files that you need to check in when you change a test file, no bizarre links across directories, no weird inter-file dependancies.

Using version control for our tests and framework meant that we were free to change stuff as we saw fit – if something didn’t work we could just revert. I can’t recommend using version control enough. The “blame” feature in svn was particularly useful ;)

Parallelization of test execution

We got our super-fast execution time by cheating: we parallelized the execution of the tests. We used a Mac Pro (16GB RAM, 8 core, 4xHDD), the Parallels virtualization software to run 12 VMs, and a shonky “n mod 12″ function to decide what tests to run on each VM (we tried to get testjour working but it needs a fair bit more dev and documentation before it works as advertised and becomes usable).

One advantage of using a Mac (apart from its rock-solid stability) was “Exposé” – we could see all 12 VMs running at once. The amount of people who walked past our desk and got hypnotized by seeing 12 tiny PCs each executing as if they were self aware… – good PR for our team!

Results sanitization

It doesn’t matter if you can run a million tests in 5 minutes, if the results aren’t accurate and sanitized quickly. We’re quite proud of the fact that we provide our results within 5 minutes of finishing the last test (usually straight away). We’ve put a lot of effort into being able to do this and it’s one of the biggest selling points of the framework we’ve written. Here’s what we’ve done to get to this stage:

  1. Though we use many VMs to execute tests on, when execution is finished we write all the results to a central location. We don’t have to worry about collating results, it’s done automatically.
  2. We use the watir checker feature heavily (see above). Any test that failed because of an environment problem is marked as such. This means that we don’t need to investigate why the test failed – we know that it didn’t get past the login screen because we saw the “database connection down” message, not because there’s a bug in login.
  3. Where we find a bug, we put the number in the relevant assertion message. If the test fails at the same point in the next run, it’s because the bug is still there. The bug will be printed to the results file and when our summary script runs over the results it’ll mark the test as a known fail due to the bug referenced in the assertion message. A particularly awesome feature!
  4. Every test that hasn’t been marked as a known fail (environment or known bug) gets added to a list of tests that need to be investigated. They’ve already been run twice (see above), so there’s a very high chance that a regression bug has crept in.
  5. As well as a count of the various flavors of failed tests, we also gave percentages. These mean far more to the scrum masters, so that’s what we gave them.

Making the effort public

Initially, there was a lot of hostility toward the project – understandably! UI testing doesn’t have a good history where I work at the moment. One of the scrum masters pointed out that the best PR was fast, consistent and accurate test results and he was right. But there are a few more things that gave people reason to change their opinion.

  1. Every time our tests found a regression bug, we updated our “bug count” on a big, visible-to-all white board. There was no denying the value that we provided (we crossed the 100 bug count in only a few weeks)
  2. We gave the scrum masters remote access to our run machine. This allowed them to kick off test runs and watch them as they progressed. The instant feedback they got blew them away. The Exposé functionality let them see environment instability issues as soon as they occurred allowing them to go and kick some heads in the infrastructure team as soon as there was a problem.
  3. We put our test execution box on the end of our desk, facing out. As mentioned above, the Exposé function was great PR for the team. In the weeks before go-live, many evenings were spent huddled around the screen as we would do test run after test run with 6 or 7 people from varying levels of management all hypnotized by the tiny windows!
  4. The test owners wrote their tests on a wiki page – as soon as a test was automated we would mark it as such. Test owners knew exactly where they stood.

Summary of the lessons we learned

  1. Sell the idea of watir-based automated UI testing with working code
  2. Manage your managers’ expectations well
  3. Keep the framework simple – complexity you don’t need introduces risk you probably can’t afford
  4. Use the right tool for the right job
  5. Use free tools where possible; you’ll be freed from the hassle of purchasing stuff
  6. Use version control – it will be your salvation many times over
  7. For object-recognition, use xpath unless you have reliable IDs you can depend on
  8. Attempt to recognize environment-related errors in the test and report on them as such (watir checkers)
  9. Keep as much of your framework code as modular and independent as possible
  10. Make the test team’s work public. Provide test results as soon as is possible.
  11. Put a lot of effort into making results sanitization as quick as possible. There is very little as frustrating for a scrum master/test manager as knowing that test execution is complete but that the results “aren’t ready yet”.

That’s all for now!

Debugging cucumber tests with ruby-debug

Testing with cucumber blows test/unit out of the water, apart from one small aspect… I can’t use the Netbeans’ GUI debugger. The Netbeans debugger has been a joy to use, but sadly I have to say goodbye. So I’m left with using the traditional ruby debugger. It uses the same approach as gdb – console based debugging. Tedious and painful. Here’s how to use it…

Preparing your cucumber project

1) Install the gem with:
gem install ruby-debug

2) In your env.rb file, add the following at the top of the file:

require 'rubygems'
require 'ruby-debug'

3) If you want to be able to see stack traces, follow the above 2 lines with:
Debugger.start
(It might be worth wrapping that in some sort of debug-mode check as it does slow things down a little.)

Now you’re ready to debug like it’s 1979!

Using ruby-debug

To get started with ruby-debug, you need to set a breakpoint. How? Like this:


def some_method
  my_obj.thing = 4
  breakpoint #look! a breakpoint!
end

Breakpoints are set by writing breakpoint. Weird, huh? But there is a gotcha – breakpoint can’t be the last line of a method, or the debugger will stop at the calling method. So, you need to do something like this:


def some_method
  my_obj.thing = 4
  breakpoint
  0
end

…which is rubbish.

Once your breakpoint is set and you run the test, you’ll eventually see a command prompt – in Netbeans it’ll be in the output window. It’s time to debug.

Here are some of the commands that are available – they cover 95% of what I usually need from a debugger.

To end the debugging session and return control to the app:
quit

To find out where you are:
where
Note, this will print out a stack trace only if you’ve added the Debugger.start as specified above. If you haven’t, where will only print out the line number and file name of where execution has reached.

To see where you are in context (current line with a few lines before and after):
list

To print out the value of an individual variable:
p my_var

To print out all local and instance variables in the current stack frame:
info variables

To go to the next step (equivalent of ‘step into’):
step

To step over the next line:
next

To let the app run to the next breakpoint:
continue

To restart the app:
restart

You can see that ruby-debug gives you pretty much everything you need. The commands listed above hardly scratch the surface of what it can do. If you want to know more, here’s a good reference: http://bashdb.sourceforge.net/ruby-debug.html

BTW, here’s a post on how to wrap the breakpoint call in a cucumber step definition.

Running NetBeans ruby tests from the command line

As far as doing UI testing with watir/firewatir goes, NetBeans is a fantastic IDE. It’s lightweight, but powerful enough to do pretty much anything I need. The integration with Test::Unit is particularly good: I select a test file, I call “Run > Test File”, I wait, I see pretty green and red indicators for the tests. But what if I want to run the tests from the command line? The NetBeans project structure means that it isn’t a matter of simply running ruby all_my_tests.rb. I took a look at what NetBeans actually did when it runs the test files (using the magic of ps -ax), and I’ve managed to compress the whole thing into:

cd project_dir
ruby -I lib -I test test/all_my_tests.rb

That’s the mac/unix/linux syntax. The windows syntax is the same:

cd project_dir
ruby -I lib -I test test\all_my_tests.rb