NEWSLETTERS
 

CIO.com updates, insights and advice on technology, management and your career.

 CIO BlackBerry News and Tips
 CIO Research and Analysis
 CIO Microsoft
 CIO Insider
 
 
 
SUBSCRIBE TO CIO
 
Are you involved in setting the direction for your company's IT budget or strategy?

Apply today for a FREE subscription to CIO Magazine!

 


Tue, Oct 27, 2009 9:35 EDT

The Seven Deadly Sins of Software Test Automation

Anonymous user

Topic: Development

Current Rating: 5 Comments: 2

by, Colin Armitage, CEO of Original Software

For the past 15-plus years, organisations have turned to test automation as a way to improve efficiency in the Software Development Life Cycle. Yet despite heavy investment, software testing is still often the bottleneck in the delivery cycle. In a recent survey of CIOs, Original Software found that only 6% were totally happy with their automation. The scary thing is that this is tolerated - It’s the norm!

“Only in a world this shitty could you even try to say these were innocent people and keep a straight face. But that's the point. We see a deadly sin on every street corner, in every home, and we tolerate it. We tolerate it because it's common, it's trivial. We tolerate it morning, noon, and night. Well, not anymore. I'm setting the example.”

John Doe (Kevin Spacey) in Se7en

Undoubtedly Test Automation has failed to achieve for the masses. But why is this the case? In this article I take a light-hearted look at Dante’s Divine Comedy to uncover some home truths about software test automation…

The Italian poet Dante Alighieri (1265-1321), wrote three epic poems (known collectively as the Divine Comedy) titled Inferno, Purgatorio, and Paradiso. In his book Inferno, Dante recounts the visions he has in a dream in which he enters and descends into hell. The sinners that Dante encounters in the Inferno are each punished in accordance with which of the seven deadly sins they were most guilty of while they were alive.

Now, I can’t help but to liken this to our very own industry –

As Dante journeys through the Inferno he encounters sinners condemned to eternal damnation because of their actions or in some cases inaction while on earth.

As I myself journey from prospect to prospect I often encounter the ‘sinners’ of test automation, whose projects are condemned to eternal damnation because of their actions, or, in many cases, the inaction that we un-earth.

One can gain a deeper understanding of Dante’s Inferno by studying the seven deadly sins which brought these souls to this miserable place.

In this article I will explore each of the seven deadly sins as they relate to software test automation, instances we come across time and time again, and traps clients have often fallen into because of their earthly vices.

Welcome to the Seven Deadly Sins of Software Test Automation.


Pride / VanityUsually considered the worst of the sins, this is a feeling of superiority and an excessive belief in a person's own abilities; A desire to be more important or attractive to others, failing to give credit due to others, or excessive love of self. This sin often manifests itself in the wanton squandering of money and time on themselves without caring about others.

Picture the scene: The wool has been pulled over the eyes of Brimstone Business Application Co’s CIO. He did a deal with a big IT supplier to have their quality-centred products added to his order. At the time it seemed like a good idea – he saved the company money and had a nice round of golf rolled-in to boot. He’d fallen into the superiority trap of believing that the most expensive or most prevalent solution would always be the best, but now was beginning to realise that this particular technology was not actually compatible with his company’s needs. He’d brought this solution in, so his pride was unable to take failure. Instead he persevered until it was too late and placed unrealistic goals on his QA team, who were then forced

You do not have flash or javascript support.
Average (1 vote)
5
 
 
Thu, Oct 29, 2009 21:56 EDT
Anonymous user
Posted by: Anonymous
Rating:

There are four parts to any automated test...the code, the script (procedure), the data an the expected results. If any piece of the test starts to change...or, in the case of data, expire...then the test case has to be revisited. I have spent more of my life than would care to admit trying to explain to upper management that you cannot automate tests if:

you don't know what the expected results are

the code is expected to change

the data is fragile, meaning that you have not designed a data set that can be reset (to avoid the obvious problems of reuising a script if you cannot add a "new" customer twice...there is validation against duplicate id numbers, etc.)

The validation in the script is so tight that the smallest change in position of a image or a field will blow up the script with a false failure.

Avoiding these problems takes very careful planning and only one of these problems (script validation) can be cured by picking a better tool.

Automation has been oversold. It is a very, very useful tool for regression of code that was not expected to change. It is considerably less useful for code currently under development.

 
Fri, Oct 30, 2009 4:43 EDT
Anonymous user
Posted by: Stephan Meyn
Rating: 70

If you want to automate testing you effectively write software. Calling it scripting is a sign of not taking this issue seriously.
In order to develop automated test cases all parts of the SDLC (or should that be TDLC) apply.
- Requirements: check.
- Architecture? that's where most people skip and pay dearly for in the long run.
- Design & Code: should benefit from design and coding standards.
- QA: you need to review these test cases, you need to test the test cases.
- Infrastructure: you have to have a proper dev environment setup.
- configuration management: just because your 300 test cases are just scripts doesn't mean you shouldn't manage that code base.

Last year we ran an automated test suite tied into a continuous build environment. We executed on average 1200 test cases a day. When you do that you also need a reporting and management infrastructure to avoid dying in the flood of data generated.

Post new comment

* Subject:
* Username:
* E-mail:
The content of this field is kept private and will not be shown publicly.
Homepage:
* Body:
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <blockquote> <strike> <p> <br>
  • Lines and paragraphs break automatically.
More information about formatting options

* Denotes required field.

Start a Conversation
Click to post

Got something to say? We want to hear it! Click the Post button to get started. GO»

EXPERT ADVICE
See our roster of experts.

Advice & Opinion from more than 113 of IT's most insightful thinkers.

  PARTNERS       WEBCASTS    
 

Windows 7 Webcast Series

There's a lot of buzz about Windows 7 out there. Each month in our webcast series, listen to analysts and customers discuss how Windows 7 and the Windows Optimized Desktop is impacting large companies around the world. Learn how they evaluated Windows 7, including the cost of deployment, deployment strategies, and tangible benefits.

Sponsored by Microsoft  Listen to on-demand Recordings »

 

Service Level Management Best Practices Life Cycle Overview - Improve Service Levels

Best practices for Service Level Management (SLM) is a process for consistently meeting customer requirements and delivering on IT's promises. See the steps required to ensure high-quality SLM.

Sponsored by Compuware  Read this White Paper »

 

Keeping Your Members Safe from Online Scams and Predators

In order to keep fraudsters out, romance sites must deploy effective solutions that look at information independent of what is supplied by users. A device fingerprinting solution such as iovation ReputationManager™ provides unique insight into the computers being used to create multiple accounts and exposes hidden device-account relationships that identity-based fraud solutions often miss.

Sponsored by iovation  Read this White Paper »

Resource Alerts

Get instant email notifications by topic when white papers, webcasts, and case studies are added to our library.

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

Defend Against Blended Threats: What You Need to Know

Blended Web and email threats are becoming increasingly complex and represent a huge...  View Now »

 

Prescriptive Actions to Reduce Risk

In this Webcast, learn best practices for effective systems management in a heterogeneous environment and keep client systems cost under control.   View Now »

 

Webcast- Vantage 11: Redefining Application Performance Management

Compuware's latest release, Vantage 11, is a major advance in end-to-end application performance management--bringing together proactive issue identification, quantification of business impact and problem resolution into a single solution. Tune in to learn how Vantage 11's top-down approach helps you make better decisions and dramatically lower operations costs.  View Now »

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

 
NEWSLETTER

Sign-up for the Blogs & Discussion Newsletter

 
FEATURED SPONSORS
 
 
 
SPONSORED LINKS
 

Return on Information: Google Enterprise Search pays you back. Get the facts.

VMware. The source for Business Infrastructure Virtualization.

ShoreTel tells businesses to untangle from competitors' complexity and turn to its brilliantly simple UC solution

See how AT&T can help protect your network.

Streamline IT Costs. Boost Performance with WAN Optimization.

Build your 1st app FREE with Force.com

TDWI checklist helps define data readiness for analytics. Download report.

A Clear View Toward Virtualization

Virtualization Technology as a Business Solution

The rules of infrastructure management just changed.

A Clear View Toward Virtualization

Interactive Q&A helps you discover key ways to maximize IT assets.

Ready to virtualize tier one applications? Check your virtualization maturity.

Think you can't afford a Cisco Switch? Cisco Catalyst Switches are now more affordable.

Five minute business analytics assessment. Immediate results.

The Case for Investing in Business Analytics Technology. Read white paper.

Upgrading to VMware vSphere with vWire

Top 10 Lessons Learned for Corporate 3G Mobile Broadband Deployments

CRM Built for IT: The Executive Guide to Selecting CRM that Meets IT Needs

Return on Information: Google Enterprise Search pays you back

ROI of Application Delivery Controllers

Making Consumer Two-Factor Authentication Simple and Cost-Effective

Mining the Cloud to Ease the Enterprise Compliance Burden

Solve Five Key IT Security Challenges with Cloud-Based Authentication

White Paper: Right-Sizing Your Power Infrastructure

AT&T Synaptic Storage as a Service. Expand on demand

Trend Micro ranked #1 against real-world malware. Read more.

Webinar: Jump-start your in-house e-discovery with Ringtail QuickCull from FTI Technology

Top Five CIO Challenges

Read the RSA report: Security for Business Innovation

64-page prescriptive guide to security, compliance, and IT operations.

Increase UPS efficiency without sacrificing protection.

eZine: A Roadmap to Reducing IT Complexity

Reduce risk, gain agility. See how Progress can help your business.

Virtualization Technology as a Business Solution

eZine: A Roadmap to Reducing IT Complexity

World-class trading technology solutions from NYSE Technologies.

If You're Paying for Telecom, You're Paying Too Much. Contact Asentinel Today.

Trade-In your old printer and save up to $1,000 plus free recycling!

infoBOOM! - The Mid-Sized Company CIO's Exclusive Community

Live Webinar: Applying Business Analytics. Click here to learn more

Removing Barriers To Better Server Virtualization Efficiency

4G Revisited. The Continued Evolution of Wireless Mobility.

What's Next for Enterprise Resource Planning?

Maximizing website Return on Information with high-quality search

Gartner Magic Quadrant, Application Delivery Controllers 2009

Authentication as a Service by Forrester Research

Cloud-Based Authentication for Next-Generation Extranets

Cut Costs & Green Your IT Operations with PC Power Management

White Paper: 4 Customer Service Myths