CUNY B r o o k l y n  C o l l e g e            Department of Computer and Information Science
CIS 763X: Software Methodology
with Dr. D. Kopec                           Fall, 2000

Back tondividual Class Presentations
Software Runaways.
Lessons Learned from Massive Software Projects Falure

by Robert L. Glass

Chapter 1 -  2.1.4

 Presented by Yin Cao

 The purpose of my presentation is to foster discussion about some software runaway war stories. When we’re failing, dig deep and think hard what’s the problems  in our runaway can effectively prevent stories like these happen again.
 

What is software runaway?

       A project that goes out of control primarily because of the difficulty of building the software needed by the system.
 

The most common definition of software crisis is:

       Software is always over budget, behind schedule, and unreliable.
 

Runaway projects could be categorized by their primary cause:

      1. Project objectives not fully specified.------------------------------51%
      2. Bad planning and estimating-----------------------------------------48%
      3. Technology new to the organization--------------------------------45%
      4.  Inadequate/No project management methodology---------------42%
      5.  Insufficient senior staff on the team--------------------------------42%
      6.  Poor performance by suppliers of hardware/software-----------42%
      7. Other---performance (efficiency) problems

Project objectives not fully specified is the main cause of software runaway
There is little doubt that project requirements are the single biggest cause of trouble on the software project front. Study after study has found that where there is a failure, requirement problems are usually found at the heart of the matter
 

      Requirement problems mostly occur when:

1.  There are far too many of them. Huge projects fail far more often than smaller ones.
2.  They are unstable. The users cannot decide what problem they really want to solve.
3.  They are ambiguous. It is not possible to determine what the requirements really mean.
4.  They are incomplete. There is insufficient information to allow a system to be built.
There were three projects that suffered most clearly from requirement trouble. They are the Denver International Airport Baggage-Handling System, the Florida Welfare System, and the FAA Air Traffic Control System.

At Denver, those responsible for the system tried to scale a relatively simple system into a far more complicated one,   there by cause a spectacular schedule overrun.

In Florida, the responsible people tried to use a centralized system from another state as the basis for a decentralized project in Florida, leading to a host of problems with increasingly ugly contractual, and political implications.

At the FAA, it was once again a mismanaged requirements story but with a different verse. The requirements for the originally proposed system were so huge that it is still unclear if such a system could be built. Only time will tell.

Next will more directly to these stories.
 

BAE AUTOMATED SYSTEMS:

DENVER INTERNATIONAL AIRPORT   BAGGAGE-HANDLING SYSTEM

The overall context of the DIA project is characterized by the dynamic nature of the development of a new airport, the economic and political environment, the large number of parties involved, the severe time constraints for completion, and the one-time nature of the project. In addition, communication and debate at the various management levels and among subcontractors was not encouraged or facilitated, in keeping with the strong emphasis on product and deadlines that characterized the fast-track nature of the project.

Two years into the construction of the new airport, a vision began to emerge for the inclusion of an airport-wide integrated baggage-handling system that could provide a major improvement in the efficiency of luggage delivery. The new airport was expected that the integrated system would improve ground time efficiency, reduce close-out time for hub operations, and decrease time-consuming manual baggage sorting and handling. The plan was to allow the airport to grow and expand without compromising efficiency.

BAE Automated systems Inc., an engineering consulting and manufacturing company     was awarded the contract.

But the result said that Denver International Airport is performing far worse than expected, and worse than the airport which DIA replaced.
 

      There were, however, a number of risks inherent in the large scale information    technology:

1.  The scale of the large project size;
2.  The enormous complexity of the expanded system;
3.  The newness of the technology;
4.  The large number of resident entities to be served by the same system;
5.   The high degree of technical and project definition uncertainty;  Short time span for completion;

The detail problems are:

1.  A Build-Design Project

BAE didn't pay enough attention to the programming issues early enough in the design process. This was a build-design project, which means that the airport was building while it was designing. There was a lot of pressure and too many players. About 200 to 300 firms and reached 400 during the construction phase, five different firms designed the runways, four the terminal.  and the program was chopped up into many small projects. Moreover, communication channels between the city, project management team, and consultants were neither well defined nor controlled. So there were at least four to six copies of everything. One of the biggest problems we had was keeping track of all the changes. It took three years to implement the tracking system.

    2.   Project Management

Over time as changes in the context and in various internal components of the DIA project took place, and the technology began to experience problems related to its newness, management was caught without the tools, time, or structures to facilitate the required changes in operational processes. As the pressure grew, different views began to appear which resulted in open conflict. The process of interaction between the management team, the system development contractor, and the airlines involved a clash of perspectives In a fast-moving, ever-changing environment such as the development of a new airport, the management structure must be able to rapidly produce engineering alternatives and the supporting cost and schedule data. But because DIA was financed by many sources and was a public works program, project administrators had to balance administrative, political, and social imperatives. The city of Denver staff and consultant team shared leadership of the project and coordinated the initial facets of DIA design. So it became evident within a very short time that doing duplicate duties which was inefficient.

3.   Implementing an Integrated Baggage-handling System need time

A grand airport like DIA needs a complex baggage system. Although the technology developed, its implementation in a complex project like DIA would have required significantly greater time than the city had left available. A united project manager concurred: ”BAE told them from the beginning that they were going to need at least one more year to get the system up and running, but no one wanted to hear that”.

5.  Project Organization Redefine

The management team never perceived the baggage-handling system as determining the economical survival of the project in any significant way, especially once the settlements with BAE took place. Furthermore, although the requirements for the system were expanded from one airline, United Airlines, to serve the whole airport, the managerial structures merely replicated past practices. The managerial team failed to
redefine the system implementation in relation to its new context..

6.  Project Relations

The relationship with the management team was very poor. The management team had  no prior baggage handling competence or experience.

7.  Politics

Two distinct processes going on, one political and the other technical, and they have little to do with one another. The city lost control at the outset, and the project was destined to run amuck. Further political problems ran through the entire Denver International Airport construction in the presence of rhetoric and false assurances to the bond market.

8.  Technologically Advanced

The BAE design is technologically advanced. It is not the next generation of baggage system, it is more like a jump from third to fifth or sixth generation. Unfortunately, BAE misused its technological advantage by expecting spectacular performance from the system components, and not allowing them a proper margin of error. The components were expected to perform to their highest theoretical capabilities.

9.  Schedule

A timetable for the opening of the airport was never realistic and should have taken potential problems into account.

         10.  System Testing

Testing the system's mechanical side was unsuccessful. Testing proved to be difficult   and more time consuming than BAE anticipated.

11.  Equipment

Early in testing, laser scanning equipment that misread bar codes became a major problem. BAE resolved some scanning difficulties by installing redundant laser scanners. Unfortunately, in BAE's case, it was difficult to pinpoint every manifestation of laser scanning error due to the number of possibilities inherent in the system.

Most of the problems related to errors in the system’s computer software, but      mechanical problems also played a part. In response to criticism after the third opening delay, BAE president explained, "We simply ran out of test time" because of changes requested by the airlines, problems "working around other vendors," and failures in the airport's electrical power supply.

The process of organization change surrounding the implementation of the  bagage-    handling system at DIA involved new ways of working, managing, and thinking.
 

 Implementation of Denver International Airport Baggage-Handling System

 Because of the problems above, the scheduled airport opening was postponed again and   again, when the airport finally opened in late February 1995, it was 16 months behind schedule and close to $2 billion over budget. So many problems were discovered that testing had to be halted. Clearly, the automated baggage system now underway at DIA is not year at a level that meets the requirements of the city, the airlines, or the traveling public. Problem arisen between the City of Denver and United Airlines around the design of the backup baggage system. United and the City of Denver unable to agree who should pay for the modifications. Integrated testing continued through the fall an winter of 1994, In January 1995, a full-scale, three-hour, 10,000 bag practice run of the substitute baggage system was completed without any problems.
 
 

FLORIDA FIASCO(FLORIDA WELFARE

Florida On - Line Recipient integrated Data Access system was supposed  save the state hundreds of millions of dollars by reducing payment errors and agency staff requirements. Instead, it has doled out hundreds of millions of dollars in overpayments, spurred mass resignations and dismissals at the state Department of Health and Rehabilitative Services(HRS), and kicked up a storm of controversy.

What’s the problem of the Florida Welfare system?

Many observers believe the FLORIDA system was flawed from the start.

HRS found human and computer error resulted in $260 million in overpayments and $58 milllion in underpayments during 1992. Complying with the federal requirement, Florida inexplicably decided to predicate its design for a distributed system on an Ohio project that used centralized computers to serve two counties. To accomplish that miracle of computer engineering, ERS proceeded to transferring several million lines cobol code from Ohio to the project in Florida. Without electricity, FLORIDA was useless to citizens whose homes and lives were brutalized. Attempts to increase FLORIDA’s capacity brought new troubles. The Medicaid overpayments in particular are directly attributable to software problems.

The FLORIDA system may also be at the root  of the human errors. FLORIDA does all eligibility calculations, which frees recipients from being shuffled from caseworker to caseworker, but also means that if FLORIDA makes a mistake, the caseworker is unlikely to catch it unless he or she has specific knowledge of the assistance program.

That still leaves FLORIDA, which limps along with its software problems unresolved. It will be fantastic, a real improvement in delivery of services.
 

         ANATOMY OF A RUNAWAY: WHAT GROUNDED THE AAS

In 1983, FAA announced plans to modernize its air traffic control system, which is a nationwide network of radars, computers, communication systems, and personnel whose mission is to safely guide aircraft throughout our air traffic system. Some of the equipment in use today has its genesis in 1950s and 1960’s technology, and some of it is still reliant on vacuum tubes.
FAA's largest single modernization project was the Advanced Automation System (AAS). The original cost estimate for AAS was $2.5 billion with a completion date of 1996. The program experienced numerous delays and cost overruns which were blamed on both FAA and the primary contractor, IBM. In 1994 FAA decided the future success of the program would be best insured if the program was redesigned. This meant canceling part of the program, splitting the program into three phases, and in some cases re-bidding the contract. The result is two independent primary contracts which provide only part of the originally promised features: one contract is with Lockheed-Martin Corporation for the en route equipment; and one is with Raytheon for the TRACON equipment.
 

Root problems in the AAS  are :

Runaway software projects are generally caused by one of the following factors: changing specifications, bad planning, new technology, bad project management, or too junior of staff.

So the first step in preventing a runaway software project is: Make sure you know what you're supposed to do. That means you need to make sure you understand the problem sufficiently, so that you know if your solution will work. At Twin Forces, that means that the first thing we do is sit down with the customer and work with him to understand the problem. The better you understand the project, the more you can make sure that you're fixing the right problem, and the more you can plan around future problems down the road.

The next step is planning the project well to begin with. Frankly, that's just experience.  We've done a lot of software projects, so we're pretty good at estimating when things will be done. Not only that, but in business its always best to underpromise and overdeliver. That is, we have the business sense to realize that we need to interface  with other people, and its much easier for people to adjust for you being early, then for  them to adjust for you being late.   New technology is always a problem, however, it doesn't have to be. The right way to integrate new technology is to integrate it gradually, a step at a time. Its rare that you really need to jump with both feet into a new technology. Generally, you can build something that works, then add a piece of the new technology. Once that new piece works, add another piece, and so on until you've fully integrated the technology. If you must add the new technology all at once, test that technology early in the project rather then later. However, to be honest, new technology is rarely a problem for us simply because we have the experience to know when and where to use new technology, not just use it because its "cool".

Bad project management and too junior of staff are generally not a problem for us because the principals are all quite experienced, and no employee has less then 6 years of experience.

 

References

1.   Robert L. Glass, “Software Runaways”, chapter 1.1-2.14
2.   Argyris, C. Overcoming Organizational Defenses", Boston: Allyn & Bacon, 1990.
3.   Ramiro Montealegre, “What Can We Learn from the Implementation of the     Automated Baggage-Handling System at the Denver International Airport?”
4.  New Denver Airport: Impact of the Delayed Baggage System (Briefing Report, 10/14/94,     GAO/RCED-95-35BR).

 
 
 
 
 
 

Comments and suggestions e-mail to Sergey D.