There are very few real requirements for a UI test automation framework:
- It should provide accurate test results
- It should provide accurate test results every time you ask it for results
- It should make it easy to write tests
- It should require little maintenance – time should be spent writing tests and analyzing results, not on coding the framework
- It should be easy to tweak in order to deal with last minute changes to the app being tested.
Projects rarely have the time or patience to deal with “we can’t run the tests just now, we’ve got a framework issue”, so frameworks tend to get built alongside the tests; and unless you’re careful, frameworks written under these (quite common) conditions normally die in one of 2 ways:
- Due to time constraints, any changes that have to be made to the test framework tend to be band-aids/hacks. “oh,-didn’t-we-tell-you-about-[insert-new-feature-that-will-break-lots-of-tests]-oh-and-can-you-kick-off-a-run-in-5-minutes?-Just-make-it-work!”, etc. That’s just the nature of the job. But, as the many dead UI frameworks that litter IT shops will attest to, there’re only so many band-aids you can stick onto a framework before it collapses under it’s own weight. Eventually, a change comes along that can’t be fixed just by “adding another band-aid” – a big refactor is required to deal with the new feature which in turn causes other framework instability problems. Test runs become unreliable resulting in the framework being abandoned.
- The other way frameworks die is when the test automation team are given time and money and are told to come back with a test automation framework… they have lots of time, so they spend lots of it on making things super-abstract, modeling business entities, writing test parsers etc. The tests that are written using the framework are all ‘semantic’, but they can’t deal with those “oh,-didn’t-we-tell-you…” changes to the app being tested. The super-abstracted nature of the framework makes it difficult to “just make it work” – there’s no one place to stick the band-aid, it needs to be spread across the framework. Many files need updating, the beautiful (but ultimately useless) business model is broken, and major refactors are required to ‘fix’ the model. During this time the tests can’t run. The framework ends up on a shelf gathering dust.
Like most things, a middle ground needs to be found:
- A framework should be flexible and simple enough to be able to deal with last minute changes in the application under test. But, small chunks of time should then be given to allow small refactors of the framework do deal with the change ‘properly’ so that the quick hack can be removed. This way, the framework stays lean and can deal with new changes on a whim.
- The framework shouldn’t be over-engineered – simplicity is key. Abstraction for abstraction’s sake is an utter waste of time. Modeling business entities in classes usually isn’t required, and when it is, only small elements of the model are usually needed for testing purposes. Doubtless, often it makes sense to model fundamental things like users, but rarely have I needed to keep track of more than the username, password and a few other simple fields. Keep business model classes simple – that way they’ll deal with application changes without much work.
Hacks for hacks’ sake aren’t good. Abstractions for abstraction’s sake aren’t good either. Write what needs to be written, don’t write what doesn’t need to be written, keep things simple, and tidy up after yourself when things get hacky.
well written, sobering and true to be sure, but still kind of leaves out the meat of the matter, which is “how do i tell if it’s over-engineered?” It seems to be a battle of extensibility vs. time. Our product is really small and would take very little framework but they keep saying it’s going to grow and grow. i’m less concerned with testing last-minute changes and more concerned with being able to test major new features.
i’ve been looking at our my new co’s half-written framework for a week deciding whether it needs to be scrapped (for some of the reasons in this and the “duct-tape programmer” article) but am now a little afraid i may be scrapping it for an equally over-engineered solution.
you freaked me out man, stop it!