One of the most common ways of structuring a series of test cases is to make one test case dependent on the outcome of another. For example, Test Case ‘A’ verifies the functionality surrounding the ability to create an account. Test Case ‘B’ verifies functionality surrounding account deletion, but instead of stating that the required data is and account in a particular state, it states that the account generated by test case ‘A’ should be the one to test for deletion. The mistake cascades through the test cycle: in execution of the test suite, if test case ‘A’ fails then test case ‘B’ cannot be executed and so it is marked as ‘failed’.
This test case interdependency causes problems for automation. It’s also a bad thing to do in general. Why?
In the above example, when it comes down to it, test case ‘B’ is not dependent on test case ‘A’ at all. If ‘B’ is testing deletion, it should test deletion. Deletion is dependent on an account, not necessarily a specific one (i.e. the one generated by test case ‘A’). OK, the account to test deletion against may need to be in a specific state (e.g. not already marked for deletion, etc…) but that’s not the same as dictating a specific account number.
As well as being, er, “philosophically” wrong, interdependency of test cases leads to testers incorrectly failing tests. Marking test case ‘B’ as failed just because test case ‘A’ did produces incorrect data in the test report. Why? Marking a test as failed when it hasn’t been executed is wrong, no matter what the reason is. The tester executing test case ‘B’ should have picked one of the (possibly) large number of valid accounts to use instead of being limited to test case ‘A’s account. That way, the ‘delete’ functionality can be tested even if the ‘create’ functionality is broken.
How is this a problem for automation? Well, an automated test should be just that: an automatic version of a manual test. Hard-wiring data into automated tests is common (and sold as a ‘feature’ of many packages), but makes the tests very fragile. If the data doesn’t exist (due to other tests failing), some tests won’t be able to run even though there may be plenty of valid data to use!
An easy fix is to make a slight modification to your tests: change them to be dependent on data in a particular state rather than specific data. Subtle difference with a large impact on test case management and execution. You’ll still be testing the same functionality, but the tests are much less interdependent. You’ll be able to execute all your tests (not just a subset) and your automated tests will be much more reliable and maintainable.
Well, I would have to say ‘yes’ and ‘no’. Interdependancy is a real problem for automation, for the reasons you give. A concatination of scripts that fail after the first, because of a functional problem encountered in the first will give a misleading impression of the state of the object under test.
In manual testing, though, the answer is a little less obvious. Firstly, it is far easier to ‘adapt’ the subsequent tests to avoid the problem encountered in the first. As a result a more accurate impression can be acquired.
Secondly, data selection can be a very real issue; so to start with a ‘known state’ can be very beneficial. Of course, if the first test fails, then all that can be known about the subsequent tests is that ‘we don’t know’.
Interdependancy also has another useful aspect in that – for the example tested – we can be 100% sure that the functional throughput performs as desired. Obviously, to be sure that full functionality is correct would usually requre a large number of interdependant tests.
The main problem with your solution is the that it is often difficult to identify with precision what the ‘particular state’ required for a test is.