Complexity, Agility and the Future of the IT Profession

The IT profession is at a turning point. One group of IT practitioners already knows what needs to be done - but the traditionalists call it radical. And the traditionalists continue to apply the same old ways of doing things that result in the same old horrendously expensive multi-year projects that produce systems barely better than what was there before, if they even work at all.

What differentiates these two groups is the manner in which they perceive and respond to complexity. A good example of this is the multi-year, multi-million dollar system infrastructure upgrade projects that so many companies are forever battling their way through. How’s your three-year systems upgrade project coming along? How many companies are going into their fifth and sixth years on those three-year projects? How many companies have spent themselves into a hole on those projects and now don’t have money to respond to the unexpected developments occurring in their markets? The reputation of the whole IT profession is dragged down by the traditionalists.

People not skilled in the use of effective techniques for dealing with complexity look at the work involved on these massive projects and become totally overwhelmed. They fall back on the use of clumsy, slow-moving, bureaucratic ways of doing things. They mumble about project management and adopt cumbersome analysis and development procedures that attempt to address all needs all at once. Analysts analyze, programmers program, documents pile up, and the years go by. Progress is gained slowly and at great expense (if there is progress at all).

An Agile and Iterative Approach

Another approach would be to respond to complexity by making rigorous use of what I believe are the six core techniques in the IT profession - joint application design; process mapping; data modeling; system prototyping; object-oriented design and programming; and system testing and rollout (I’ve discussed them in previous posts such as “Six Techniques Lie at the Heart of IT Agility [1]”).

We can use these techniques to break up big problems and reduce complexity into manageable, self-contained pieces. We can use these techniques to deliver value fast by making progress right away on the simpler pieces of a problem and building solutions to the more complex pieces iteratively over time.

A reader of this blog (who does not wish to be named but agreed to have his ideas shared here – we’ll call him Carl) contacted me a while back with a story about the hard times the IRS has gone through trying to upgrade their systems infrastructure (here's an insightful article on that [2] by Roger Sessions). Developing a suite of systems to process the tax returns of individuals and corporations has to be one of the most complex jobs I can imagine. Business processes may be complex, but they can’t hold a candle to processes born of politics and government regulations. How could anyone ever make headway on a project like this?

Carl had this to say [italics are mine], “Here’s how I believe the IRS systems should have been done and the same is true of most large multi-business case applications.” He laid out an approach that I realized immediately was a great way to solve a problem like this. His approach displays elegant simplicity in the face of apparently overwhelming complexity. It is elegant in the way it uses just a handful of techniques to manage this complexity. He said, “Process each [tax payer’s] account in its own thread – not whole files – it greatly reduces complexity and latency and increases scalability. Use pipe-and-filter with no intervening files instead of [the traditional approach involving] file-process-file-process-etc. This architecture was described by Mary Shaw at Carnegie Mellon years ago.”

Think Big, Start Small, and Deliver Quickly

Carl went on to say, “Design and write a separate thread for each business case beginning with the most common or simplest, e.g. domestic-exempt-salaried-employee. The IRS could have been processing all simple 1040 returns in a matter of months on the new system instead of taking years trying to develop monolithic, all-case, highly complex systems. In my scenario there might be 10 or 30 or more separately designed and written processes handling business case threads.” In other words, quickly deliver a “robust 80% solution” and then keep expanding on it over time.

Do a web search on “pipe-and-filter [3]” and read some of the references that come up. It uses combinations of four core techniques: joint application design; process mapping; object-oriented design/programming; and system testing/rollout. Any IT practitioner skilled in the use of these techniques will quickly grasp the main concepts of the pipe and filter approach. And there are plenty of other approaches like this that use combinations of the core techniques to effectively deal with complexity. How come we don’t use them? Why do we continue with massive, multi-year, multi-zillion dollar projects? How much longer do we need to keep doing these kinds of projects and hoping they will somehow turn out differently?

Every successful profession has to develop and learn to use a set of core techniques that enable its practitioners to succeed most of the time. We in the IT profession can allow ourselves to be intimidated by complexity and cling to ineffectual, bureaucratic approaches that give us a reputation as a bunch of not-so-lovable screw-ups; or we can get agile and reinvent ourselves. We can become practitioners of a profession respected for its ability to deal with complexity and apply technology effectively in demanding situations.

So which way will IT go as a profession?

[Michael Hugos, principal at Center for Systems Innovation [c4si], [4] delivers seminars and briefings and mentors teams in agile systems development and strategies for business agility. His newest book is Business Agility: Sustainable Prosperity in a Relentlessly Competitive World.]