Rants
Questions
Soapbox
Best Practices
Apply today for a FREE subscription to CIO Magazine!
Tue, Oct 27, 2009 9:35 EDT
|
Posted by: Anonymous in Best Practices Topic: Development
Current Rating: |
by, Colin Armitage, CEO of Original Software
For the past 15-plus years, organisations have turned to test automation as a way to improve efficiency in the Software Development Life Cycle. Yet despite heavy investment, software testing is still often the bottleneck in the delivery cycle. In a recent survey of CIOs, Original Software found that only 6% were totally happy with their automation. The scary thing is that this is tolerated - It’s the norm!
“Only in a world this shitty could you even try to say these were innocent people and keep a straight face. But that's the point. We see a deadly sin on every street corner, in every home, and we tolerate it. We tolerate it because it's common, it's trivial. We tolerate it morning, noon, and night. Well, not anymore. I'm setting the example.”
John Doe (Kevin Spacey) in Se7en
Undoubtedly Test Automation has failed to achieve for the masses. But why is this the case? In this article I take a light-hearted look at Dante’s Divine Comedy to uncover some home truths about software test automation…
The Italian poet Dante Alighieri (1265-1321), wrote three epic poems (known collectively as the Divine Comedy) titled Inferno, Purgatorio, and Paradiso. In his book Inferno, Dante recounts the visions he has in a dream in which he enters and descends into hell. The sinners that Dante encounters in the Inferno are each punished in accordance with which of the seven deadly sins they were most guilty of while they were alive.
Now, I can’t help but to liken this to our very own industry –
As Dante journeys through the Inferno he encounters sinners condemned to eternal damnation because of their actions or in some cases inaction while on earth.
As I myself journey from prospect to prospect I often encounter the ‘sinners’ of test automation, whose projects are condemned to eternal damnation because of their actions, or, in many cases, the inaction that we un-earth.
One can gain a deeper understanding of Dante’s Inferno by studying the seven deadly sins which brought these souls to this miserable place.
In this article I will explore each of the seven deadly sins as they relate to software test automation, instances we come across time and time again, and traps clients have often fallen into because of their earthly vices.
Welcome to the Seven Deadly Sins of Software Test Automation.

Pride / Vanity – Usually considered the worst of the sins, this is a feeling of superiority and an excessive belief in a person's own abilities; A desire to be more important or attractive to others, failing to give credit due to others, or excessive love of self. This sin often manifests itself in the wanton squandering of money and time on themselves without caring about others.
Picture the scene: The wool has been pulled over the eyes of Brimstone Business Application Co’s CIO. He did a deal with a big IT supplier to have their quality-centred products added to his order. At the time it seemed like a good idea – he saved the company money and had a nice round of golf rolled-in to boot. He’d fallen into the superiority trap of believing that the most expensive or most prevalent solution would always be the best, but now was beginning to realise that this particular technology was not actually compatible with his company’s needs. He’d brought this solution in, so his pride was unable to take failure. Instead he persevered until it was too late and placed unrealistic goals on his QA team, who were then forced
There are four parts to any automated test...the code, the script (procedure), the data an the expected results. If any piece of the test starts to change...or, in the case of data, expire...then the test case has to be revisited. I have spent more of my life than would care to admit trying to explain to upper management that you cannot automate tests if:
you don't know what the expected results are
the code is expected to change
the data is fragile, meaning that you have not designed a data set that can be reset (to avoid the obvious problems of reuising a script if you cannot add a "new" customer twice...there is validation against duplicate id numbers, etc.)
The validation in the script is so tight that the smallest change in position of a image or a field will blow up the script with a false failure.
Avoiding these problems takes very careful planning and only one of these problems (script validation) can be cured by picking a better tool.
Automation has been oversold. It is a very, very useful tool for regression of code that was not expected to change. It is considerably less useful for code currently under development.
If you want to automate testing you effectively write software. Calling it scripting is a sign of not taking this issue seriously.
In order to develop automated test cases all parts of the SDLC (or should that be TDLC) apply.
- Requirements: check.
- Architecture? that's where most people skip and pay dearly for in the long run.
- Design & Code: should benefit from design and coding standards.
- QA: you need to review these test cases, you need to test the test cases.
- Infrastructure: you have to have a proper dev environment setup.
- configuration management: just because your 300 test cases are just scripts doesn't mean you shouldn't manage that code base.
Last year we ran an automated test suite tied into a continuous build environment. We executed on average 1200 test cases a day. When you do that you also need a reporting and management infrastructure to avoid dying in the flood of data generated.