The Chaos Factory

"In late 1988, Business Week published an article that stated a “survey of 600 firms indicated that 35% of them had at least one ‘runaway’ software project”[1]. Soon after, Dr. Barry Boehm, the Director of the U.S. Department of Defense’s Advanced Research Projects Agency published a series of articles about software risk management which further raised awareness about failed software projects.

The Project Management Institute (PMI) had been working very hard to elevate project management to a profession with standards and practices that could be considered equal to that of engineering. As part of this effort, studies were instituted on projects of all sorts, and for the next few years reports started emerging with statistics on software development project success/failure rates. They were not encouraging[2].

Finally, two very influential reports came out in 1994: The Standish Group’s Chaos Report[3], and KPMG’s Report on IT Runaway Systems[4], each of which independently surveyed large companies with large IT projects (in the US and the UK respectively). Both reported that in the opinion of the executives surveyed, barely a third of their IT projects could be considered definite successes. The failure of in-house custom software development projects was becoming well documented and corporate executives, especially executives of public corporations, were confronted with the responsibility of having to seek solutions to prevent the waste of shareholder’s money (estimated by both Standish and the PMI to be in the billions) as auditors increasingly focused on IT spending.

There were four different responses to this burgeoning crisis from four different communities: computer scientists (and software engineers), project managers, IT managers, and business managers. Over the next few chapters I will tell you about each one.

The academics, computer scientists, and software engineers naturally took scientific approaches: gathering data, proposing hypo­theses, and then testing them through experimentation. The result of this was an explosion of proposed methodologies claiming to address the root causes of IT project failures. We will go into these in the very next chapter.

The project managers took the most practical approach possible, building on the existing de facto process (known as the waterfall method and also discussed in the next chapter) to address the concerns of the auditors and the finance department. It bears remembering that many corporate IT departments were still reporting to the CFO at this time. Project management also has a chapter to itself, Chapter Eight.

IT managers reacted defensively. The Chaos Report‘s first paragraph referred to a paper co-authored by the computer scientist Alfred Spector[5].

“In 1986, Alfred Spector, president of Transarc Corporation, co-authored a paper comparing bridge building to software development. The premise: Bridges are normally built on-time, on- budget, and do not fall down.

….

One of the biggest reasons bridges come in on-time, on-budget and do not fall down is because of the extreme detail of design. The design is frozen and the contractor has little flexibility in changing the specifications. “

IT managers did not have to be told twice, and the quest to nail down specifications became an absolute mania. To my dismay, IT managers rushed to become “the department of No”, often taking a truculently defiant stance that no estimates would be issued, no commitments made, until the internal business client signed in blood that the requirements were set in stone and complete.

I saw this as a fast track to marginalization and insignificance because as the Chaos Report also pointed out on the same page:

“…in today’s fast moving business environment, a frozen design does not accommodate changes in the business practice”.

Commercial Off the Shelf

I was not wrong. The nineties became a time of unprecedented friction between IT and the business units within companies, and this brings us to the response of the business managers and auditors.

Just as freezing requirements became an obsession for IT managers, bypassing IT completely became a fixation for many business managers. With the success of the PC and shrink-wrapped software they figured they had the tools to do it. COTS became the buzzword of the day. It is an acronym for Commercial Off The Shelf, and the idea was that eliminating internal development projects would eliminate project failures. Let a third party assume the risk went the thinking. If they fail to deliver, then we don’t pay them. The company is protected. This philosophy was very closely aligned to the overall business trend towards outsourcing and suited the purchasing department and the auditors just fine, thank you very much. No less a personage than Fred Brooks endorsed the idea. For a number of years at the end of the nineties and the early years of the millennium, the policy of purchasing COTS instead of opting for internal program development was anointed best practice and pursued aggressively.

This trend hit its peak during the time I was CTO of a large international organization. Every other week, I was meeting with an external software vendor who had been brought in by a business manager in an attempt to do an end run around IT only to be brought up short when they discovered a need to integrate with in-house systems.

Unlike many other IT managers, I was not hostile to these vendors. I tried to be tough but fair. I was fundamentally sympathetic to the business users’ desire to cut me out. I was the representative of a department that seemed to take a perverse delight in thwarting even the slightest progress, patiently lecturing in the most condescending of tones that business needed “to understand” this, that or the other thing, or “that’s just not the way it works,” or just like the building contractor tells you about your kitchen renovations, “you can have it fast, good, or cheap, but not all three”.

James H. Boren’s delightful adage from his legendary work When in Doubt, Mumble is:

“Bureaucracy is the epoxy that greases the wheels of progress,”[6]

I used to joke that I was an international bureaucrat, and it was my job to prevent things from happening. But in reality, I deplored the attempts of my colleagues to make business conform to the limits that IT lacked the courage to surpass. I felt, as I always have and still do, that technology has no relevance if it is not used in service of some human pursuit such as the pursuit of business. I wanted my department to push whatever technology or process boundaries it needed to in order to keep up with the demands of the market and the need of the business to stay competitive.

I successfully pushed for a reorganization that moved IT Business Solutions into a separate team from that of the glass house so we could be free of their foot-dragging and no-but-isms. I deliberately had the IT Business Solutions team act just as a third-party software vendor would. I bid competitively against the external vendors the business managers brought in. I tried to win on merit, not hegemony. I placed my IT Business Solution Managers in the departments they served and drilled them to never respond to an idea with the type of passive-aggressive statements that the other IT managers insisted their team members use. Phrases such as “It might be very difficult…” and “it could be very complicated” and the infamous “you have to understand…” were replaced by “that sounds like a good idea, let’s see what it would take to make that work” or “I think I understand why you want to go in that direction. Can I just play it back to you to make sure I do?”

But my efforts notwithstanding, the best practices of the purchasing department and auditors saw to it that for several years the COTS vendors did very well, selling systems to business managers who signed contracts in blissful ignorance of the fate that awaited them.

The ultimate outcome of the COTS fad was tragically predictable because large public companies all have two things in common.

First, all of them have entirely unique, byzantine processes that are sacred and totally unchangeable because of corporate culture. Large companies have strange ways of doing things. One large company I worked at (a multibillion-dollar corporation) took Gross Revenues, deducted only commissions (not any of the other costs) and called it Net Revenue – not exactly what the rest of the world understands by that term. My favorite was the organization that created Northern Asia and Southern Asia as two territories and then declared Singapore to be in Northern Asia. So mighty were we, that even the earth’s geography would bend to our will. These are just a few small examples of the separate reality that large corporations live in.

Second, they all evaluate expenditures as being driven by one of only two possible motivations: They make investments either to get ahead of the competition, or so as not to get left behind by the industry.

When an innovation occurs (technical or any other kind), a few adventurous, risk-tolerant executives will seize on it to get ahead of the competition. Therefore, they want to build in-house to keep things proprietary. This is a relatively cost-insensitive endeavor. As long as the cost is a reasonable percentage of the revenues it generates, no further attention is paid. Growth is more important than cost control.

When enough other companies have copied the innovation, it is no longer a competitive advantage and the adventurous executives completely lose interest in it. Companies that join the trend late benefit from low prices because third-party vendors will have entered the market making it a commodity. The conservative, risk averse executives that inherit responsibility for the early in-house innovations feel pressure to abandon expensive proprietary implementa­tions in order to bring their costs down to the same level as that enjoyed by the late comers. They will then seek a cookie cutter solution (ideally maintained by a third party) at the lowest possible cost. All innovations suffer this fate; the line is constantly being redrawn. Today’s precious proprietary technology is tomorrow’s boat anchor. Like some sharks, companies have to keep moving or they will suffocate.

The COTS strategy is suitable for these late-stage situations where a technology is no longer a competitive advantage. Everyone is happy with a cookie cutter solution as long as it keeps costs to a bare minimum.

However, for dealing with unique corporate culture and processes, or the desire for proprietary competitive features, COTS is an utter failure. The whole premise of COTS and the basis of its success is that it is cookie cutter. Trying to customize it is a recipe for disaster, and it is a recipe that serves generous portions. Disasters with COTS customization projects became legendary, even more prone to spectacular failure than internal custom development.

COTS fell out of vogue on the heels of a string of highly publicized multi-million dollar failures such as the 1993 shipping and inventory catastrophes that brought drug company FoxMeyer and appliance maker Whirlpool to bankruptcy court with SAP, and the dire shortage of Hershey’s Kisses and Jolly Ranchers on the shelves for Halloween 1999 after Hershey tried to implement COTS software from SAP, Seibel, and Manugistics.

After a number of these debacles, there was simply no way that anyone could conceivably continue to call it best practice to buy COTS unless the intention was to use it exactly as-is. This coincided with some very significant changes to accounting rules, and between the two things, in-house custom development suddenly came very much back into fashion.

[1] Rothfeder, Jeffrey, It’s Late, Costly, Incompetent – But Try Firing a Computer System. Business Week, November 7, 1988.

[2] The Project Management Institute estimated that North American companies lost “nearly $300 billion on late, overbudget, or failed implementations during 1999–2001”

[3] Standish Group International. The Chaos Report; www.standishgroup.com/sample_ research/PDFpages/Chaos1994.pdf.

[4] KPMG. Report on IT Runaway Systems. [Online]. Strategys for KPMG Management Consulting, 1994. [16/02/2017]. Available from: eis.mdx.ac.uk/research/SFC/Reports/KPMG_ITsystems.pdf

[5] https://www.projectsmart.co.uk/white-papers/chaos-report.pdf

[6] Boren, James H. When In Doubt, Mumble. New York [Etc.], Van Nostrand Reinhold, 1972.