NEWSLETTERS
 

CIO.com updates, insights and advice on technology, management and your career.

 CIO BlackBerry News and Tips
 CIO Research and Analysis
 CIO Microsoft
 CIO Insider
 
 
 
SUBSCRIBE TO CIO
 
Are you involved in setting the direction for your company's IT budget or strategy?

Apply today for a FREE subscription to CIO Magazine!

 


Tue, Oct 27, 2009 9:35 EDT

The Seven Deadly Sins of Software Test Automation

Anonymous user

Topic: Development

Current Rating: 5 Comments: 2

by, Colin Armitage, CEO of Original Software

For the past 15-plus years, organisations have turned to test automation as a way to improve efficiency in the Software Development Life Cycle. Yet despite heavy investment, software testing is still often the bottleneck in the delivery cycle. In a recent survey of CIOs, Original Software found that only 6% were totally happy with their automation. The scary thing is that this is tolerated - It’s the norm!

“Only in a world this shitty could you even try to say these were innocent people and keep a straight face. But that's the point. We see a deadly sin on every street corner, in every home, and we tolerate it. We tolerate it because it's common, it's trivial. We tolerate it morning, noon, and night. Well, not anymore. I'm setting the example.”

John Doe (Kevin Spacey) in Se7en

Undoubtedly Test Automation has failed to achieve for the masses. But why is this the case? In this article I take a light-hearted look at Dante’s Divine Comedy to uncover some home truths about software test automation…

The Italian poet Dante Alighieri (1265-1321), wrote three epic poems (known collectively as the Divine Comedy) titled Inferno, Purgatorio, and Paradiso. In his book Inferno, Dante recounts the visions he has in a dream in which he enters and descends into hell. The sinners that Dante encounters in the Inferno are each punished in accordance with which of the seven deadly sins they were most guilty of while they were alive.

Now, I can’t help but to liken this to our very own industry –

As Dante journeys through the Inferno he encounters sinners condemned to eternal damnation because of their actions or in some cases inaction while on earth.

As I myself journey from prospect to prospect I often encounter the ‘sinners’ of test automation, whose projects are condemned to eternal damnation because of their actions, or, in many cases, the inaction that we un-earth.

One can gain a deeper understanding of Dante’s Inferno by studying the seven deadly sins which brought these souls to this miserable place.

In this article I will explore each of the seven deadly sins as they relate to software test automation, instances we come across time and time again, and traps clients have often fallen into because of their earthly vices.

Welcome to the Seven Deadly Sins of Software Test Automation.


Pride / VanityUsually considered the worst of the sins, this is a feeling of superiority and an excessive belief in a person's own abilities; A desire to be more important or attractive to others, failing to give credit due to others, or excessive love of self. This sin often manifests itself in the wanton squandering of money and time on themselves without caring about others.

Picture the scene: The wool has been pulled over the eyes of Brimstone Business Application Co’s CIO. He did a deal with a big IT supplier to have their quality-centred products added to his order. At the time it seemed like a good idea – he saved the company money and had a nice round of golf rolled-in to boot. He’d fallen into the superiority trap of believing that the most expensive or most prevalent solution would always be the best, but now was beginning to realise that this particular technology was not actually compatible with his company’s needs. He’d brought this solution in, so his pride was unable to take failure. Instead he persevered until it was too late and placed unrealistic goals on his QA team, who were then forced

You do not have flash or javascript support.
Average (1 vote)
5
 
 
Thu, Oct 29, 2009 21:56 EDT
Anonymous user
Posted by: Anonymous
Rating:

There are four parts to any automated test...the code, the script (procedure), the data an the expected results. If any piece of the test starts to change...or, in the case of data, expire...then the test case has to be revisited. I have spent more of my life than would care to admit trying to explain to upper management that you cannot automate tests if:

you don't know what the expected results are

the code is expected to change

the data is fragile, meaning that you have not designed a data set that can be reset (to avoid the obvious problems of reuising a script if you cannot add a "new" customer twice...there is validation against duplicate id numbers, etc.)

The validation in the script is so tight that the smallest change in position of a image or a field will blow up the script with a false failure.

Avoiding these problems takes very careful planning and only one of these problems (script validation) can be cured by picking a better tool.

Automation has been oversold. It is a very, very useful tool for regression of code that was not expected to change. It is considerably less useful for code currently under development.

 
Fri, Oct 30, 2009 4:43 EDT
Anonymous user
Posted by: Stephan Meyn
Rating: 70

If you want to automate testing you effectively write software. Calling it scripting is a sign of not taking this issue seriously.
In order to develop automated test cases all parts of the SDLC (or should that be TDLC) apply.
- Requirements: check.
- Architecture? that's where most people skip and pay dearly for in the long run.
- Design & Code: should benefit from design and coding standards.
- QA: you need to review these test cases, you need to test the test cases.
- Infrastructure: you have to have a proper dev environment setup.
- configuration management: just because your 300 test cases are just scripts doesn't mean you shouldn't manage that code base.

Last year we ran an automated test suite tied into a continuous build environment. We executed on average 1200 test cases a day. When you do that you also need a reporting and management infrastructure to avoid dying in the flood of data generated.

Post new comment

* Subject:
* Username:
* E-mail:
The content of this field is kept private and will not be shown publicly.
Homepage:
* Body:
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <blockquote> <strike> <p> <br>
  • Lines and paragraphs break automatically.
More information about formatting options

* Denotes required field.

Hot Conversations

Ex-Microsofties Look Back in Anger

Posted by Shane ONeill in News | 4 comments

The Price of IT Outsourcing

Posted by Beth Bacheldor in Best Practices | 2 comments

Start a Conversation
Click to post

Got something to say? We want to hear it! Click the Post button to get started. GO»

EXPERT ADVICE
See our roster of experts.

Advice & Opinion from more than 115 of IT's most insightful thinkers.

  PARTNERS       WEBCASTS    
 

Windows 7 Webcast Series

There's a lot of buzz about Windows 7 out there. Each month in our webcast series, listen to analysts and customers discuss how Windows 7 and the Windows Optimized Desktop is impacting large companies around the world. Learn how they evaluated Windows 7, including the cost of deployment, deployment strategies, and tangible benefits.

Sponsored by Microsoft  Listen to on-demand Recordings »

 

A Framework for Better Application Delivery

The complexity of application delivery is driven in part by the evolving applications environment. Instead of approaching application delivery from a siloed fashion, this handbook looks at end-to-end guidance and discusses the impact of ignoring the WAN, Web apps that are chatty, data center consolidation, SaaS, Web 2.0 and virtualization.

Sponsored by Riverbed  Read this White Paper »

 

Microsoft® Exchange 2010 includes archiving - but is it enough?

Microsoft® Exchange 2010 includes basic email archiving. But many organizations will find that it does not meet their requirements. This paper describes why organizations need to archive, what capabilities Exchange 2010 includes and why 3rd party archiving solutions will be necessary for most organizations.

Sponsored by Google, Inc.   Read this White Paper »

Resource Alerts

Get instant email notifications by topic when white papers, webcasts, and case studies are added to our library.

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

Enterprise Capture: Your Onramp to Business Process Automation

Today more than ever companies are seeking to reduce costs and...  View Now »

 

The True Cost of Legacy Systems

How well are you maximizing existing software assets? This webcast reveals the results of a commissioned study on top migration and modernization priorities for IT leaders.   View Now »

 

How To Maximize Your Virtualization Strategy and Deployment

Join award-winning technology journalist Stan Gibson in this webcast as he discusses how to enhance your virtualization strategy with the ROI, planning, implementation and platform advice. Exploit the business benefits of virtualization and successfully expand your current deployment.   View Now »

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

 
NEWSLETTER

Sign-up for the Blogs & Discussion Newsletter




*Required fields

By clicking the sign-up button, you agree to the Privacy Policy.

View all newsletters »

 
FEATURED SPONSORS
 
 
 
SPONSORED LINKS
 

Manage limitless content todayread EMCs 15-minute guide to ECM.

HP Exstream. Get a Free Document Assessment for Financial Services.

Take the Netezza TwinFin TestDrive!

Webinar: Jump-start your in-house e-discovery with Ringtail QuickCull from FTI Technology

Let Progress Software help your business make progress.

Best Practices to Reduce IT Operational Costs

Real-world testing ranks Trend Micro #1 against malware. See results.

Forrester: The real-world financial impact of Windows 7

Turn your desk phone and mobile phone into one with Sprint Mobile Integration.

Maximizing efficiencies with unified communications.

Stay informed with custom newsletters from Tech Dispenser

Selecting the Right Reporting Technology

An IT Leadership Action Plan for the Economic Recovery

Consolidate data centers and lower IT service costs. Learn How.

WAN optimization techniques significantly improve application performance. Read More.

The Revolution and Evolution of Private Cloud Computing

ROI of Application Delivery Controllers

Cut Costs & Green Your IT Operations with PC Power Management

Enterprise Capture: Your Onramp to Business Process Automation

Adobe® LiveCycle®solutions for intuitive user experience

Unlocking the Mainframe: Modernizing Legacy System to SOA

State of the Data Integration Market

Enhance Customer Loyalty through Higher Responsiveness

Achieving Business Agility with Application Grid

Seven Ways ITIL Can Help You in an Economic Downturn

Does your IDS really work? Find out with a free Endace Audit

Verint Systems. Discover the Power of Intelligence in Action"

CA ARCserve r12.5 is More Than Backup! Download Trial Version Today

Enterprise search helps employees get more done. Get the facts from Google.

See why ShoreTel is named best overall VoIP provider by Nemertes Research

Trend Micro ranked #1 against real-world malware. Read more.

AT&T Application Management & Hosting. Let us help you STRETCH

Microsofts new client operating system helped Pella reduce power consumption.

Efficiency goes up. Costs come down.

Dark Fiber from Sunesys Save on Unlimited Bandwidth with Fixed Costs.

Trend Micro ranked #1 against real-world malware. Read more.

Webcast: Solve Your Data Visualization Needs with Open Source BI

Webcast: Delivering the Enterprise-Ready Cloud

Ensure cost effective application delivery. Learn More.

Cloud Computing: The Impact CIOs See

What's Next for Enterprise Resource Planning?

Gartner Magic Quadrant, Application Delivery Controllers 2009

Global Research: CIOs Weigh In On Virtualization

Adobe® LiveCycle® solutions for business process automation

What's New in SOA Suite 11g?

Unleash the Power of Java with Oracle JRockit Real Time

SOA Best Practices and Design Patterns

Application Grid: Ideal Platform for IT Consolidation

Taking the Service Desk to the Next Level