Tag Archives: testing

Finding the balance between hacky and over-engineered UI test automation frameworks

Posted on by
1

There are very few real requirements for a UI test automation framework:

  1. It should provide accurate test results
  2. It should provide accurate test results every time you ask it for results
  3. It should make it easy to write tests
  4. It should require little maintenance – time should be spent writing tests and analyzing results, not on coding the framework
  5. It should be easy to tweak in order to deal with last minute changes to the app being tested.

Projects rarely have the time or patience to deal with “we can’t run the tests just now, we’ve got a framework issue”, so frameworks tend to get built alongside the tests; and unless you’re careful, frameworks written under these (quite common) conditions normally die in one of 2 ways:

  1. Due to time constraints, any changes that have to be made to the test framework tend to be band-aids/hacks. “oh,-didn’t-we-tell-you-about-[insert-new-feature-that-will-break-lots-of-tests]-oh-and-can-you-kick-off-a-run-in-5-minutes?-Just-make-it-work!”, etc. That’s just the nature of the job. But, as the many dead UI frameworks that litter IT shops will attest to, there’re only so many band-aids you can stick onto a framework before it collapses under it’s own weight. Eventually, a change comes along that can’t be fixed just by “adding another band-aid” – a big refactor is required to deal with the new feature which in turn causes other framework instability problems. Test runs become unreliable resulting in the framework being abandoned.
  2. The other way frameworks die is when the test automation team are given time and money and are told to come back with a test automation framework… they have lots of time, so they spend lots of it on making things super-abstract, modeling business entities, writing test parsers etc. The tests that are written using the framework are all ‘semantic’, but they can’t deal with those “oh,-didn’t-we-tell-you…” changes to the app being tested. The super-abstracted nature of the framework makes it difficult to “just make it work” – there’s no one place to stick the band-aid, it needs to be spread across the framework. Many files need updating, the beautiful (but ultimately useless) business model is broken, and major refactors are required to ‘fix’ the model. During this time the tests can’t run. The framework ends up on a shelf gathering dust.

Like most things, a middle ground needs to be found:

  • A framework should be flexible and simple enough to be able to deal with last minute changes in the application under test. But, small chunks of time should then be given to allow small refactors of the framework do deal with the change ‘properly’ so that the quick hack can be removed. This way, the framework stays lean and can deal with new changes on a whim.
  • The framework shouldn’t be over-engineered – simplicity is key. Abstraction for abstraction’s sake is an utter waste of time. Modeling business entities in classes usually isn’t required, and when it is, only small elements of the model are usually needed for testing purposes. Doubtless, often it makes sense to model fundamental things like users, but rarely have I needed to keep track of more than the username, password and a few other simple fields. Keep business model classes simple – that way they’ll deal with application changes without much work.

Hacks for hacks’ sake aren’t good. Abstractions for abstraction’s sake aren’t good either. Write what needs to be written, don’t write what doesn’t need to be written, keep things simple, and tidy up after yourself when things get hacky.

Running NetBeans ruby tests from the command line

Posted on by
Reply

As far as doing UI testing with watir/firewatir goes, NetBeans is a fantastic IDE. It’s lightweight, but powerful enough to do pretty much anything I need. The integration with Test::Unit is particularly good: I select a test file, I call “Run > Test File”, I wait, I see pretty green and red indicators for the tests. But what if I want to run the tests from the command line? The NetBeans project structure means that it isn’t a matter of simply running ruby all_my_tests.rb. I took a look at what NetBeans actually did when it runs the test files (using the magic of ps -ax), and I’ve managed to compress the whole thing into:

cd project_dir
ruby -I lib -I test test/all_my_tests.rb

That’s the mac/unix/linux syntax. The windows syntax is the same:

cd project_dir
ruby -I lib -I test test\all_my_tests.rb

Fitnesse, Ruby and the Mac

Posted on by
Reply

I’ve recently started using Fitnesse, an acceptance testing framework. There really isn’t much around on how to get it working on the mac with ruby, so I decided to get it working and put up my findings…

Installing the Fit Ruby Gem

First of all, you’ll need to download the Fitnesse ruby gem. It isn’t complete by any stretch of the imagination, but it does the job. To get it, run the following command, entering your administrator password when prompted:


sudo gem install fit

You should see the following as a result:


Successfully installed fit-1.1

Installing the Fitnesse Server

Now, you’ll need to download Fitness… Click here, and select the most recent version (20070619 at time of writing). I downloaded the fitnesse20070619.zip version.

Unzip the file, and copy the resulting fitnesse folder to the Applications folder (there’s where I put it… if you do the same, it’ll be easy to follow along with this post).

Allowing Fitnesse to run

We now have to set the correct unix file permissions to allow the Fitnesse server to run. Back to the terminal, this time entering the following commands:


cd /Applications/fitnesse/
chmod +x run.sh

That sets the “I can be executed” flag on the file, allowing us to run the server. Not very Mac-like, but it has to be done.

Setting up the Ruby Fixture Code

We’ll now prepare the ruby fixture code (ie: the test) that we’ll eventually be executing. Having this in place before we create the Fitnesse page gives us the ability to test everything we do whilst trying to get it all to work – it’s a painful process, anything to make it easier is a good thing).

I created the following file…

/Users/nat/Development/Projects/Ruby/WatirFitFramework/lib/Framework/Testcase.rb

…with the following contents…

require 'rubygems'
require 'fit/column_fixture'
module Framework
class Testcase < Fit::ColumnFixture
attr_accessor :numerator
attr_accessor :denominator
def quotient
@numerator.to_f / @denominator.to_f
end
end
end

Make sure that the the case used for the name is consistent. I advise using a filename beginning with a capital letter, followed by only lowercase letters.

Setting up the Fitnesse Test Page

We first of all need to start the fitnesse server. Back to the terminal…


cd /Applications/fitnesse/
./run.sh -p 8080

The “-p 8080″ means that the fitnesse server will be running on port 8080. Ports lower than 1024 will cause errors.

Now open Safari (or any other web browser – I’m using safari for this post) and navigate to http://127.0.0.1:8080. If you see the “Welcome to Fitnesse!” page, all is good.

Due to the way that test suites work in fitnesse, I decided to put the config stuff in a parent page, and the test itself in a child page. I haven’t got the space/time/energy to explain that concept – see the fitnesse documentation for that.

I created the “NatTest” page by going to http://127.0.0.1:8080/NatTest and clicking on the “create this page” link that is displayed as a result.

In the page (click the ‘Edit’ link in the navigation column) I added the following:


!define COMMAND_PATTERN {ruby -I %p /Library/Ruby/Gems/1.8/gems/fit-1.1/bin/FitServer.rb}
!path /Users/nat/Development/Projects/Ruby/WatirFitFramework/lib

The first line tells fitnesse how to run ruby, and the second line tells fitnesse where our ruby fixture code is.

Next, I created the actual fitnesse test by navigating to http://127.0.0.1:8080/NatTest.TheTest, again clicking on the “create this page” link. In it, I pasted the following ColumnFixture table:


|Framework.Testcase|
|numerator|denominator|quotient?|
|10|2|5|
|12.6|3|4.2|
|100|4|24|

Last thing now… we need to make our “TheTest” page an actual test so that fitnesse can execute it. To do that, click on the “Properties” link in the navigation column, check the “Test” check button and then click “Save properties”.

Run the test

In the navigation column there should now be a “Test” button. Click it…

If all has gone well, you should see some test results. From here on, you’re by yourself.