NEWSLETTERS
 

CIO.com updates, insights and advice on technology, management and your career.

 CIO BlackBerry News and Tips
 CIO Research and Analysis
 CIO Microsoft
 CIO Insider
 
 
 
SUBSCRIBE TO CIO
 
Are you involved in setting the direction for your company's IT budget or strategy?

Apply today for a FREE subscription to CIO Magazine!

 


Tue, Nov 20, 2007 20:21 EST

Why the Waterfall "Failed"?

Topic: Applications

Current Rating: 4 Comments: 11

Thoughts on applicability of different development methodologies

May be it’s not a waterfall that failed, but our understanding of it as Dr. Kruchten pointed out (see http://www.ibm.com/developerworks/rational/library/4626.html). However, waterfall clearly has its limitations. I tend to think that we tried to use it beyond its boundaries of applicability and now blame the process while we really need to look in the mirror. Let’s consider the topic a bit closer.

There are a great many tasks that are based on rigid and stable algorithms. It is possible – and natural – to come up with “correct” upfront requirements and design for these tasks. This means that it is possible to apply the serialized software development process that we used to call “waterfall”. The last statement has at least two corollaries. Firstly, linear / serialized development methods are not dead. And what’s more, they should be used for the said subset of tasks. Let’s call them deterministic tasks. Secondly, if we cannot “do it right” from the first attempt for the “right” type of tasks, this says something about our abilities to do our job and nothing else.

Most business applications belong to a different subset of tasks, however. Although each program is based on an algorithm anyway, these business tasks cannot be reduced to stable, once-and-forever described algorithms. We can call these tasks partially-deterministic.

I have been involved in developing business applications since the mid-70s. Never ever during my entire career has a client (a user, customer, stakeholder, etc.) known what he wants. Even less likely was that could he distinguish between “wants” and “needs”, let alone to envision what he will need tomorrow.

Why is this? Are they all stupid? Or may be they are just evil and like to torture us, poor software developers? I tend to say that the massive scale of this phenomenon calls for a quest for less extravagant reasons. We may even suggest that we are dealing with the nature of a beast. Here are two “less extravagant” reasons that I like the most. The first one is more or less obvious: they (clients) deal with a very volatile reality, which changes on them every day. Indeed, market conditions, human behaviours, acts of competitors and governments, social processes cannot be reduced to stable algorithms.

I think the second reason is not so obvious. Here it is: not only does the volatile reality influence the requirements and hence our systems, but other way around is true too. Like in O. Henry’s famous novel, the systems we develop influence the reality! This is a very important point that we do not always remember. The extent of our influence on reality is yet to be calculated. Future scientists may come up with an indicator similar to Planck’s constant to define a minimal system change beyond which we influence the reality that we are trying to automate.

So, certain tasks call for a certain process. If the above two thoughts are correct, then it’s a nail in the coffin of the linear process’ applicability to non-algorithmic tasks. Similarly, however, using an iterative process for a highly algorithmic and well-defined task may not be the best approach either.

You do not have flash or javascript support.
Average (3 votes)
4
 
 
Wed, Nov 21, 2007 10:03 EST
Anonymous user
Posted by: AJ Burke
Rating: 70

I like the terms used here of 'deterministic' and 'partially deterministic' and concur that not only do we see the 'variable reality' as perceived by the end-user of computerized systems we develop, but that those systems can actually alter that reality...

 
Tue, Nov 27, 2007 19:08 EST
Posted by: mtruxaw
Rating: 90

I agree with you that theoretically there may be some "highly algorithmic and well defined task" where a H2O-Fall may work as well as an agile approach. I don't know if I have ever actually participated in such a project, but I suppose it is possible. Nevertheless, I can't see the downside to using an agile, iterative approach even on these types of projects. What is the down side for delivering the highest priority portions of that system earliest? What is the downside to getting customer eyes on the product earlier to make sure your "highly algorithmic and well-defined task" was not misinterpreted by the coders. Give me a good agile iterative approach any day. Small deliverables early and often, inspection and adaptation, etc

 
Wed, Nov 28, 2007 11:34 EST
Anonymous user
Posted by: Anonymous
Rating: 30

Very good treatise that on System influences reality. I have been ranting same here on system altering the user behavior. I wish this concept gets mainstream and gets tackled better.

Now back to Waterfall model, I do not believe there is any problem in trying to determine as much as you can before embarking on design and development. The problem is trying to implement everything in one go. This takes away our ability to learn from mistakes and apply corrective actions.

 
Thu, Nov 29, 2007 10:01 EST
Anonymous user
Posted by: Kupe
Rating:

I am in agreement. You have to first understand the problem or business need you are trying to address. Once that is determined it makes perfect sense to implement the solution in an agile or iterative matter.

 
Mon, Dec 3, 2007 18:46 EST
Posted by: Eugene Nizker
Rating:

“…I can't see the downside to using an agile…”, one of my readers said. This is one of the reasons why I wrote the article in the first place.

As we know, Agile is a mindset. It cannot be reduced to a number of techniques. However, we often hear: “Use X, and it’ll solve all of your problems”. No, it won’t! “You cannot be agile unless you do Y and Z”. Yes, you can! When an approach that is based on common sense is reduced to a set of religious beliefs, it loses common sense. And then everyone loses – us and our clients.

When we get excited by something – a new book, a new friend, or a new software development methodology – it’s not easy to step back and look at our new “acquisition” with a critical eye. Yet, everything has its limitations, its boundaries. Remember, Newton’s physics didn’t die with Einstein’s breakthrough discovery. But it’s boundaries of applicability were redefined. We still happily use it every day without thinking about
relativistic effects.

What are the boundaries of applicability for a development process, be it waterfall or Agile? “Give me a good agile iterative approach any day” may be an expensive proposition. Every release requires regression testing. Auto-tests do help, but in their turn require maintenance, ever more with every new test. Add to it preparation and conduct of a promotion (often risky), potential disruption of operations and user re-training. All in all, I can see how Agile approach can become more expensive than Waterfall for straightforward algorithmic tasks with stable requirements.

As I said in my article, I have not seen this kind of projects yet during my 30+ years in software development. But that doesn’t mean they do not exist. After all, I only worked in the enterprise applications field.

It would be a stretch to paint me as an adept of Waterfall. As a member of Vancouver Agile council, and a leader of a team that practiced Agile for a few years with impressive results, I am sold on Agile. One can see my previous posts in this blog: “Agile Software Development - A Cinderella Story?”, “How One CIO was Converted to Agile”, “A Client as a Teammate” and others. However, I see Agile becoming more and more of a religion, and this is a very unfortunate development.

Regards,

Eugene Nizker

PS. A friend of mine who taught me how to write meaningful code some 30 years ago, Alex Zakharyonok, commented on another thought in my article: the systems we develop influence the reality. Alex says: “For business applications, I would formulate it more aggressively: we should only develop systems that change the reality; otherwise they aren’t worth our time”. I can’t agree more.

Post new comment

* Subject:
* Username:
* E-mail:
The content of this field is kept private and will not be shown publicly.
Homepage:
* Body:
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <blockquote> <strike> <p> <br>
  • Lines and paragraphs break automatically.
More information about formatting options

* Denotes required field.

Start a Conversation
Click to post

Got something to say? We want to hear it! Click the Post button to get started. GO»

EXPERT ADVICE
See our roster of experts.

Advice & Opinion from more than 113 of IT's most insightful thinkers.

  PARTNERS       WEBCASTS    
 

Windows 7 Webcast Series

There's a lot of buzz about Windows 7 out there. Each month in our webcast series, listen to analysts and customers discuss how Windows 7 and the Windows Optimized Desktop is impacting large companies around the world. Learn how they evaluated Windows 7, including the cost of deployment, deployment strategies, and tangible benefits.

Sponsored by Microsoft  Listen to on-demand Recordings »

 

Service Level Management Best Practices Life Cycle Overview - Improve Service Levels

Best practices for Service Level Management (SLM) is a process for consistently meeting customer requirements and delivering on IT's promises. See the steps required to ensure high-quality SLM.

Sponsored by Compuware  Read this White Paper »

 

Keeping Your Members Safe from Online Scams and Predators

In order to keep fraudsters out, romance sites must deploy effective solutions that look at information independent of what is supplied by users. A device fingerprinting solution such as iovation ReputationManager™ provides unique insight into the computers being used to create multiple accounts and exposes hidden device-account relationships that identity-based fraud solutions often miss.

Sponsored by iovation  Read this White Paper »

Resource Alerts

Get instant email notifications by topic when white papers, webcasts, and case studies are added to our library.

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

Defend Against Blended Threats: What You Need to Know

Blended Web and email threats are becoming increasingly complex and represent a huge...  View Now »

 

Prescriptive Actions to Reduce Risk

In this Webcast, learn best practices for effective systems management in a heterogeneous environment and keep client systems cost under control.   View Now »

 

Webcast- Vantage 11: Redefining Application Performance Management

Compuware's latest release, Vantage 11, is a major advance in end-to-end application performance management--bringing together proactive issue identification, quantification of business impact and problem resolution into a single solution. Tune in to learn how Vantage 11's top-down approach helps you make better decisions and dramatically lower operations costs.  View Now »

Resource Alerts

Get instant email notification when white papers, webcasts, and case studies are added to our library. Don't just be up-to-date—be up to the minute with our new Resource Alerts.

 
NEWSLETTER

Sign-up for the Blogs & Discussion Newsletter

 
FEATURED SPONSORS
 
 
 
SPONSORED LINKS
 

Return on Information: Google Enterprise Search pays you back. Get the facts.

VMware. The source for Business Infrastructure Virtualization.

ShoreTel tells businesses to untangle from competitors' complexity and turn to its brilliantly simple UC solution

See how AT&T can help protect your network.

Streamline IT Costs. Boost Performance with WAN Optimization.

Build your 1st app FREE with Force.com

TDWI checklist helps define data readiness for analytics. Download report.

A Clear View Toward Virtualization

Virtualization Technology as a Business Solution

The rules of infrastructure management just changed.

A Clear View Toward Virtualization

Interactive Q&A helps you discover key ways to maximize IT assets.

Ready to virtualize tier one applications? Check your virtualization maturity.

Think you can't afford a Cisco Switch? Cisco Catalyst Switches are now more affordable.

Five minute business analytics assessment. Immediate results.

The Case for Investing in Business Analytics Technology. Read white paper.

Upgrading to VMware vSphere with vWire

Top 10 Lessons Learned for Corporate 3G Mobile Broadband Deployments

CRM Built for IT: The Executive Guide to Selecting CRM that Meets IT Needs

Return on Information: Google Enterprise Search pays you back

ROI of Application Delivery Controllers

Making Consumer Two-Factor Authentication Simple and Cost-Effective

Mining the Cloud to Ease the Enterprise Compliance Burden

Solve Five Key IT Security Challenges with Cloud-Based Authentication

White Paper: Right-Sizing Your Power Infrastructure

AT&T Synaptic Storage as a Service. Expand on demand

Trend Micro ranked #1 against real-world malware. Read more.

Webinar: Jump-start your in-house e-discovery with Ringtail QuickCull from FTI Technology

Top Five CIO Challenges

Read the RSA report: Security for Business Innovation

64-page prescriptive guide to security, compliance, and IT operations.

Increase UPS efficiency without sacrificing protection.

eZine: A Roadmap to Reducing IT Complexity

Reduce risk, gain agility. See how Progress can help your business.

Virtualization Technology as a Business Solution

eZine: A Roadmap to Reducing IT Complexity

World-class trading technology solutions from NYSE Technologies.

If You're Paying for Telecom, You're Paying Too Much. Contact Asentinel Today.

Trade-In your old printer and save up to $1,000 plus free recycling!

infoBOOM! - The Mid-Sized Company CIO's Exclusive Community

Live Webinar: Applying Business Analytics. Click here to learn more

Removing Barriers To Better Server Virtualization Efficiency

4G Revisited. The Continued Evolution of Wireless Mobility.

What's Next for Enterprise Resource Planning?

Maximizing website Return on Information with high-quality search

Gartner Magic Quadrant, Application Delivery Controllers 2009

Authentication as a Service by Forrester Research

Cloud-Based Authentication for Next-Generation Extranets

Cut Costs & Green Your IT Operations with PC Power Management

White Paper: 4 Customer Service Myths