Announcing ‘bewildr’ – test your WPF UI apps with IronRuby

Get it here: http://www.bewildr.info

After writing an automation framework to test a WPF GUI using IronRuby and White, I decided to write a ruby-specific gem for automating WPF UI tests. Kinda like Watir, but for WPF instead of the web. The gem is called bewildr - I’ve been working on it for a while and it’s finally in a releasable state.

Here’s an example of bewildr being used in rspec to show you what it’s all about:

require 'rubygems'
require 'spec'
require 'bewildr'

describe "my example app" do
  it "should not allow invalid users to log in" do
    #start the app and wait for the main window
    @app, @main_window = Bewildr::Application.start_app_and_wait_for_window("c:\\app.exe", /App v1.\d+/)

    username_field = @main_window.get(:id => "username")
    password_field = @main_window.get(:id => "password")
    login_button   = @main_window.get(:type => :button, :name => "Go")

    #some initial checks...
    username_field.should be_enabled
    password_field.should be_a_password_field

    #attempt login with invalid user
    username_field.text = "invalidUser"
    password_field.text = "s3cr3t"
    login_button.click

    #check we're not logged in
    @main_window.get(:id => "login_message").text.should match("Wrong username/password")
  end
end

And, since bewildr was written ‘BDD-style’ there are loads of examples of its use in cucumber here: http://github.com/natritmeyer/bewildr/tree/master/features/

Cool:

  • It’s written in ruby
  • It’s free (as in speech – BSD license)
  • It’s free (as in beer – there are no bazillion-dollar yearly license fees)
  • It’s easy to install (here’s how: gem install bewildr)
  • It has a clean API (makes for idiomatic tests in cucumber/rspec)
  • It has a strong test focus (…of the testers, by the testers, for the testers…)
  • It’s been written BDD-style
  • It’s updated frequently
  • It allows tests to be written in an interpreted language (with all the flexibility which that gives you – unlike White)

Not Cool:

  • It’s limited to IronRuby, not MRI/YARV (it needs access to .Net automation)
  • It’s new – expect bugs (please raise them when you find them)
  • It’s built on top of MS UI Automation (expect quirkiness)
  • There is quite a bit to do before it does everything I want it to, eg: it doesn’t yet allow you to test for visibility, there’s no drag-and-drop; see here for more

Note: I have no intention to support anything but WPF. No WinForms, no Silverlight, no [insert non-WPF tech here]. Not yet anyway…

So, if you’re looking for a UI testing tool to automate your WPF app with, but QTP is too expensive and White is too [insert pejorative here]… Well, what are you waiting for?  http://www.bewildr.info

Testing a website on different versions of IE

So, no matter how much you argue that it’s an ancient, irrelevant browser; there’s no way you can wriggle out of having to test your web app against IE6 on WinXP. But… trying to find a machine with it lying around might be difficult. Magnanimous Microsoft have made testing your web app on different IE/Windows combinations less tedious than it could be: combine Virtual PC with a collection of pre-built images and you’re on your way. Here are the details…

First, you’ll need to install Virtual PC:

  • Download the latest version of Virtual PC from here if you’re running Windows 7
  • Download Virtual PC 2007 from here if you’re running Windows XP/Vista

Next, you’ll need to download as many of the following combinations of Windows/IE as you want from here.

The Windows/IE combinations available are:

  • IE6 / XP SP3
  • IE7 / XP SP3
  • IE7 / Vista SP1
  • IE7 / Vista SP2
  • IE7 / Vista SP3
  • IE8 / XP SP3
  • IE8 / Vista SP1
  • IE8 / Vista SP2
  • IE8 / Vista SP3

Download the images you need, start them, get your testing done before the VM runs out of time (they’re time limited) and then get back to doing something less painful!

Enjoy your multi-IE-version testing. Rather you than me ;)

Explaining Watir, Selenium and WebDriver

Something I seem to be explaining to people all the time… the relationships between Watir, Watir “2.0″, Selenium, Selenium “2.0″ and WebDriver.

Alister Scott has done an excellent job of explaining the impact of WebDriver on Selenium and Watir and what it mean for their respective futures. Read it here.

Summary:

  • Selenium 1 => dead
  • Selenium 2 == WebDriver
  • Watir 1.* (+ derivs) => dead
  • Watir 2.0 => Watir API using WebDriver to communicate with the browser

===Update===
Added version number to Watir to avoid the confusion seen here.

Testing redirects with ruby

—UPDATE—
Since writing this post, I’ve put together a gem called ‘responsalizr‘ which is a way better solution than what follows in this post. Read about it here. And now back to the original post…

Testing redirects from a web app is simple enough – make a request and check the response code making sure it’s a 301, 302 or whatever you’re expecting. The test you end up writing isn’t nice idiomatic ruby though. So, I wrote a quick monkey patch… here it is:

class Net::HTTPResponse
  #returns true if the response is a 200
  def ok?
    instance_of?(Net::HTTPOK)
  end

  #returns true if the response is a 301
  def found?
    instance_of?(Net::HTTPFound)
  end

  #returns true if the response is a 302
  def moved_permanently?
    instance_of?(Net::HTTPMovedPermanently)
  end

  #returns true for any kind of redirect (301..307)
  def redirect?
    kind_of?(Net::HTTPRedirection)
  end

  #returns the url being redirected to if this response is a redirect
  def redirect_url
    redirect? ? self['location'] : raise("Not a redirect response")
  end
end

Basically, it adds the following predicate methods to the Net::HTTPResponse class: “ok?” (returns true if response code is 200), “found?” (returns true if response code is 301), “moved_permanently?” (returns true if response code is 302) and “redirect?” (returns true if the response is a redirect – from 301..307). Because these are predicate methods, they can be used by rspec – your test code suddenly becomes much cleaner! The moneky patch also adds the “redirect_url” method which returns the url to be redirected to if the response is a redirect of some sort. The advantage of this is that your tests become much more idiomatic:


#the urls we're testing with...
@url_initially_navigated_to = "http://mail.google.com"
@expected_redirect_url = "https://www.google.com/.../etc/..."

#make the request
@response= Net::HTTP.get_response(URI.parse(@url_initially_navigated_to))

#nice idiomatic tests! - use the one you're expecting...
@response.should be_ok                #expecting a 200
@response.should be_found             #expecting a 301
@response.should be_moved_permanently #expecting a 302
@response.should be_a_redirect        #expecting anything between 301..307

#testing the url being redirected to
@response.redirect_url.should match(@expected_redirect_url)

There you go! Nice idiomatic redirect tests! Enjoy.

Remove junk from IronRuby cucumber output

When running cucumber tests under IronRuby, your output will be full of junk like the following:


Feature: example feature

Scenario: a scenario←[90m     # features\testoutput.feature:3←[0m
←[32mGiven some test setup←[90m  # features/step_definitions/output_steps.rb:1←[0m←[0m
←[32mWhen I do something←[90m    # features/step_definitions/output_steps.rb:5←[0m←[0m
←[32mThen something happens←[90m # features/step_definitions/output_steps.rb:9←[0m←[0m

1 scenario (←[32m1 passed←[0m)
3 steps (←[32m3 passed←[0m)
0m0.160s

This is not great. All those “[32m” things make the output difficult to read and cause (me at least) some serious distraction. Thankfully, it’s easily fixed… if you add the –no-color option when you call the icucumber command, ie:

icucumber --no-color

…the output will change to be:


Feature: example feature

Scenario: a scenario     # features\testoutput.feature:3
Given some test setup  # features/step_definitions/output_steps.rb:1
When I do something    # features/step_definitions/output_steps.rb:5
Then something happens # features/step_definitions/output_steps.rb:9

1 scenario (1 passed)
3 steps (3 passed)
0m0.260s

Much better, and easily solved!

RQuery: A thin layer of JQuery over watir/selenium

So I went to a presentation today, run by Aidy Lewis and Josh Chisholm about Josh’s new RQuery project; here’s a braindump of what I remember.

RQuery is not supposed to be a competitor to Selenium or Watir but builds on top of it to provide a few things… the ones that interested me were the ability to query for objects by using css selectors, and the fact that it converts ruby into javascript so that very little work is done inside the browser.

The css selector stuff is done by injecting jQuery into the page that’s being tested (if it’s not already there). This is pretty cool because it means you don’t need to worry about browser specific ways to select stuff (useful, since watir and selenium are/are going to be sat on top of webdriver so browser agnosticism is important). It also means that you don’t need to know xpath – cool if you’re a webdev since you don’t want to learn something new, and you probably already know css!

The ruby-to-javascript stuff also looks pretty cool: instead of running loads of ugly javascript in the browser, the ruby code in your test is converted to some simple jQuery and that is what gets executed. Interesting to note that in the demo we saw, the tester (sorry, didn’t catch your name!) mentioned that the only downside to this is when debugging – did the error occur in ruby? In the ruby-javascript translation? In the CSS? In the browser? In the AUT?

Check out the rQuery wiki for how to get started.

Overall, RQuery looks like a cool project. I’ll be following it for sure…

How to test a WPF app using IronRuby and White

UPDATE
Since I wrote this post, I have put together a ruby gem designed for testing WPF UIs called ‘bewildr’. I wrote up an introductory post about bewildr here. Bewildr removes the need for using White – it’s written in ruby, for ruby! You’ll find its API clean and idiomatic. Anyway, back to the original post…
—————–

…and you thought ruby was only good for web testing…

So… having previously failed at getting ruby-based automated testing of WPF working, I attacked it from a different angle and succeeded! The difference is that this time I used IronRuby instead of ruby. In doing so, the White Automation library could be used out-of-the-box! It’s time to throw away your expensive, proprietary test tools and replace them with open source tools instead! Here’s how it’s done:

Prerequisites

  • Install .Net Framework 2.0, 3.0 (and possibly 3.5 – I’ve got it installed and haven’t tested this stuff without it)
  • Install IronRuby (but put it in c:\ironruby, not the default location that contains spaces in the path)
  • Download the latest version of White (0.19 at time of writing)
  • Install UISpy.exe to investigate objects on screen
  • Bookmark the White API – you’ll use this a lot

I’m presuming that you know to run ruby files in IronRuby using the ir command, not the standard ruby command, iirb instead of irb and irake instead of rake.

Requires, Dlls and some helpful methods

First thing, you need to gain access to White in your project. To do that, put the following at the top of your app (eg: your env.rb file in cucumber):


$LOAD_PATH << File.expand_path("lib/White_Bin_0.19")
require "White.Core.dll"
require 'UIAutomationTypes, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

module IronRubyHelpers
 def r_array_to_cs_array(args)
   System::Array[System::String].new(args.map {|arg| arg.to_s.to_clr_string})
 end

 def r_string_to_c_string(r_string)
   System::String.new(r_string)
 end
end

This requires some explanation… The first line puts the directory containing the White.Core.dll file on the load path. In this case, I’ve taken the White_Bin_0.19 directory that was created when I expanded the zip I downloaded from the White site, and put it inside the lib directory in my NetBeans project. The next line requires the White library. The line after that loads the UIAutomationTypes.dll – you’ll need this if you want to find WPF objects by their type.

Next, the IronRubyHelpers module… it contains 2 methods: one converts a ruby array to a dot net array (required whenever a White object method takes an array as an argument, eg: menu interaction), and the other converts a ruby string to a dot net string (sometimes required when a White object arg takes a string).

That’s all the setup you need.

Starting and Killing WPF Apps

Now we start dealing with White… Here’s how to start a WPF app :

@app = White::Core::Application.Launch("c:\my_app\example.exe")

And here’s how to kill it:

@app.kill

The Launch(string) method is one of many ways to start an app. Check out the Application.cs documentation to see how else it can be done (I can’t get AttachOrLaunch to work – ymmv).

Grabbing Windows

You get access to objects by using the window that they’re in. That means you need to get the window first. Here’s how it’s done:

@main_window = @app.get_windows.select{|window| window.title == "My Window Title"}.first

That will get you a window whose title matches “My Window Title” exactly. If you want a bit more flexibility (eg: if your app displays its version number in the title and you don’t want to have to change your recognition string everytime you get a new build), you can use the following regex-powered window finder:

@main_window = @app.get_windows.select{|window| window.title =~ /My App, version (.*)/}.first

Your window takes a while to appear? Here’s how to wait for it:

Timeout.timeout(10) do
  sleep 0.2 until @app.get_windows.select{|window| window.title == "My App"}.size > 0
end

Now you’ll wait for up to 10 seconds checking ~5 times a second to see if a window entitled “My App” appears (it only takes a tiny tweak to make it use regex). If it takes longer than 10 seconds you’ll get a Timeout exception.

Grabbing and Interacting with Objects

Now that we can get hold of windows, we can look inside them and get access to the object that they contain. White uses static methods on SearchCriteria to access object. Generally, you’ll need 2 bits of info:

  1. How to search (eg: by id, by control type, by text, etc)
  2. The value to search for (eg: “btnLogin”, ControlType.Edit, “Login…”)

Here are some examples:

@chk_visibility = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByAutomationId(r_string_to_c_string("visibleCheckBox")))

Hey, no one said it was pretty. Here we’re getting a checkbox that lives in the main window. We’re identifying it by its automation id, which happens to be “visibleCheckBox”. Another example:

@cmb_zoom = @main_window.get(White::Core::UIItems::Finders::SearchCriteria.ByControlType(System::Windows::Automation::ControlType.ComboBox))

Here we’re getting the only combo box in the main window. This time we’re searching by ControlType. You can find a full list of the available control types here under the “Fields” section.

There are a few more ways to use SearchCriteria, you can read about them in the SearchCriteria.cs documentation.

And now to object interaction… Once you’ve got your object reference, you can call methods on it and hope it responds. Here are some examples:

@chk_visibility.checked

…will return true or false based on whether the checkbox is checked – read about the available methods on checkbox here. Another example:

@cmb_zoom.select(r_string_to_c_string("200%"))

This will select the “200%” option from the combo box.

Depending on the object type you could call .click, .value = "hello", .double_click, etc.

Navigating the Window Menu

Since using the window menu doesn’t fit the standard object-locator pattern, we’ll cover that here.

def select_menu_item(*args)
 my_menu_bar = @main_window.menu_bar
 my_menu_item = my_menu_bar.menu_item(r_array_to_cs_array(args))
 my_menu_item.click
end

select_menu_item("File", "Open...")

Using window menus is a bit weird in White. The easiest way to go about it is to use the select_menu_item above. It gets the menu bar out of the window, does some ruby-to-dot-net magic with the arguments and then clicks the menu item. I’ve included an example call to the method that does the standard File->Open action.

A few more things

Because we’re using IronRuby, there’s a quirk you need to be aware of – you may have spotted it already… IronRuby translates White’s CamelCase methods to the standard ruby_underscore_case. Eg:, if the White documentation says that a particular object supports the “DoubleClick” method, you can call that method in your ruby scripts with “double_click”. You get used to it.

Speed and Reliability

…or lack of same. White is slow. And buggy. IronRuby startup time is slow too. I’ve found loads of places where White claims to do something (eg: table interaction), but what’s there isn’t unusable. Some of the stuff that does work only works often, not all the time. I’ve rewritten some of the buggy functionality using the low level interaction libraries provided by microsoft and the replacement code runs several orders of magnitude faster than white. Please don’e expect fast execution – you just won’t get it. Be prepared for molasses-speed testing.

Example project using Paint.NET

I’ve put together a badly designed set of tests around Paint.NET (3.5.3 at time of writing) using Test::Unit. It’s a NetBeans project, but even so you’ll need to run the tests off the command line (there’s no IronRuby-NetBeans support that I can find). Make sure to have Paint.NET installed in its default location and then try running “irake test” in the project directory… and have patience – it crawls. It takes around 2 minutes to run 4 tests.

>>> Download my example project here <<<

Enjoy!