Software for Dependable Systems Sufficient Evidence? (2007) / Chapter Skim
Currently Skimming:
1 Assessment
1 Assessment
Pages 16-50
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 16... ...
Software dependability is a pressing concern for several reasons: • Developing software to meet existing dependability criteria is notoriously difficult and expensive. Large software projects fail at a rate far higher than other engineering projects, and the cost of projects that deliver highly dependable software is often exorbitant.
|
From page 17... ...
Department of Transportation's Office of the Inspector General and the Government Accountability Office track the progress of all major FAA acquisition projects intended to modernize and add new capabilities to the National Airspace System. As of May 2005, of 16 major acquisition projects being tracked, 11 were over budget, with total cost growth greater than $5.6 billion; 9 had experienced schedule delays ranging from 2 to 12 years; and 2 had been deferred.3 Software is cited as the primary reason for these problems.
|
From page 18... ...
It must not be forgotten that creating dependable software systems itself has economic consequences. Consider areas such as dynamic routing in air traffic control, where there are not only significant opportunities to improve efficiency and (arguably)
|
From page 19... ...
For example, as noted elsewhere, in the summer of 2005, radiotherapy machines in Merseyside, England, and in Boston were attacked by computer viruses. It makes little sense to invest effort in ensuring the dependability of a system while ignoring the possibility of security vulnerabilities.
|
From page 20... ...
According to the EPA, 17 123 chemical plants in the United States could each expose more than a million people if a chemical release occurred, and a newspaper article reports that a plant in Tennessee gave a worst-case estimate of 60,000 people facing death or serious injury from a vapor cloud formed by an 13 For more information on TCAS, see the FAA's "TCAS home page." Available online at |
From page 21... ...
Available online at |
From page 22... ...
An early report in the subsequent investigation noted, however, that an action was available to the pilots that would have restored power, but it was not shown on the user interface due to its position on a list, and a software design that would have required items higher on the list to be manually cleared in order for that available action to be shown.24 Perhaps the most serious software-related near miss incident to date occurred on September 14, 2004. A software system at the Los Angeles Air Route Traffic Control Center in Palmdale, California, failed, preventing any voice communication between controllers and aircraft.
|
From page 23... ...
were found to be due to faulty software.28 Of these, 192 -- almost 80 percent -- were caused by defects introduced during software maintenance.29 The actual incidence of failures in medical devices due to software is probably much higher than these numbers suggest, as evidenced by a GAO study30 that found extensive underreporting of medical device failures in general. 26William H
|
From page 24... ...
Doyle, 2001, "Patient safety, potential adverse drug events, and medical device design: A human factors engineering approach," Journal of Biomedical Informatics 34(4)
|
From page 25... ...
Moreover, the integration of invasive devices with hospital networks will ultimately erase the gap between devices and databases, so that failures in seemingly unimportant back-office applications might compromise patient safety. Networking also makes hospital systems vulnerable to security attacks; in the summer of 2005, radiotherapy machines in Merseyside, England36 were attacked by a computer virus.
|
From page 26... ...
Although the computerization of health care can offer improvements in safety and efficiency, care is needed so that computerization does not undermine the safety of existing manual procedures. In the medical device industry, for example, while many of the largest manufacturers have well-established safety programs, smaller companies may face challenges with respect to safety, perhaps because they lack the necessary resources and expertise.39 Infrastructure By enhancing communication and live data analysis, software offers opportunities for efficiency improvements in transportation and other infrastructure.
|
From page 27... ...
, software failures can be sudden and unexpected and, due to coupling, can have far-reaching effects. In 2005, for example, Toyota identified a software flaw that caused Prius hybrid cars to stall or shut down when traveling at high speed; 23,900 vehicles were affected.42 In the realm of communications infrastructure, advances in telecommunications have resulted in lower costs, greater flexibility, and huge increases in bandwidth.
|
From page 28... ...
Distribution of Energy and goods Software failures could also interrupt the distribution of goods and services, such as gasoline, food, and electricity. An extended blackout during wintertime in a cold area of the United States would be an emergency.
|
From page 29... ...
But there are few grounds for confidence, and some of the most widely used electronic voting software has been found by independent researchers to be insecure and of low quality.50 In the 2006 election in Sarasota County, Florida, the outcome was decided by a margin of 363 votes, yet over 18,000 ballots cast on electronic voting machines did not register a vote. A lawsuit filed to force a revote cites, among other things, the possibility of software malfunction and alleges that the machines were improperly certified.51 PRObLEMS WITH ExISTINg CERTIFICATION SCHEMES Evidence for the efficacy of existing certification schemes is hard to come by.
|
From page 30... ...
On the other hand, in the domain of software security, certification has been a dismal failure: New security vulnerabilities appear daily, and certification schemes are regarded by developers as burdensome and ineffective. Security Certification Security certification standards for software were developed initially in response to the needs of the military for multilevel-secure products that could protect classified information from disclosure.
|
From page 31... ...
And because the certification process at economically feasible evaluation levels focuses on the functioning of the product's security features even while real vulnerabilities can occur in any component or interface, real-world vulnerability 53 The smartcard industry has embraced higher levels of evaluation, and many smartcard products have completed evaluation at EAL 5. Of more than 400 evaluated products other than smartcards listed at |
From page 32... ...
The evaluation was useful insofar as it demonstrated the operating system contained a relatively complete set of security features, However, Microsoft's assessment was that the vulnerability rate of Windows Server 2003 was better than that of Windows 2000 because of a reduced incidence of errors at the coding level, a level well below the level at which it is scrutinized by the CC evaluation. Another example is a recent comparison57 of the vulnerability rates of database products, which indicated that a product 55 See, for example, the National Vulnerability Database online at |
From page 33... ...
The problem with CC goes beyond the certification process itself. Its fundamental assumption is that security certification should focus on security components -- namely, components that implement security features, such as access control.
|
From page 34... ...
At least in comparison with other domains (such as medical devices) , avionics software appears to have fared well inasmuch as major losses of life and severe injuries have been avoided.
|
From page 35... ...
does not significantly increase the probability of detecting any serious defects that remain in the software. Medical Software Certification Medical software, in contrast to avionics software, is generally not subject to uniform standards and certification.
|
From page 36... ...
Without better methods for developing dependable software, it may not be possible to build the systems we would like to build. When software is introduced into critical settings, the benefits must obviously outweigh the risks, and without convincing evidence that the risk of catastrophic failure is sufficiently low, society may be reluctant to field the system whatever the benefits may be.
|
From page 37... ...
68 See, for example, the NASA Aviation Safety Program. Available online at |
From page 38... ...
How many accidents can be attributed to software failures? Which development methods are most cost-effective in delivering dependable software?
|
From page 39... ...
The lack of systematic reporting of significant software failures is a serious problem that hinders evaluation of the risks and costs of software failure and measurement of the effectiveness of new policies or interventions. In traditional engineering disciplines, the value of learning from failure is well understood,73 and one could argue that without this feedback loop, software engineering cannot properly claim to be an engineer 72 See The Risks Digest, a forum on risks to the public in computers and related systems moderated by Peter G
|
From page 40... ...
First, it has informed the key notions that evidence be at the core of dependable software development, that data collection efforts are needed, and that transparency and openness be encouraged so that those deploying software in critical applications are aware of the limits of evidence for its dependability and can make fully informed decisions about whether the benefits of deployment outweigh the residual risks. Second, it has tempered the committee's desire to provide prescriptive guidance -- that is, the approach recommended by the committee is largely free of endorsements or criticisms of particular development approaches, tools, or techniques.
|
From page 41... ...
The therapists who operated the radiotherapy system that failed in Panama, for example, were blamed for entering data incorrectly, even though the system had an egregious design flaw that permitted the entry of invalid data without generating a warning, and they were later tried in court for criminal negligence.75 In several avionics incidents, pilots were blamed for issuing incorrect commands, even though researchers recognized that the systems themselves were to blame for creating "mode confusion." 76 Understanding software failures demands a systems perspective, in which the software is viewed as one component of many, working in concert with other components -- be they physical devices, human operators, or other computer systems -- to achieve the desired effect. Such a perspective underlies the approach recommended in Chapter 3.
|
From page 42... ...
might be made. Traditional software development approaches use specification and design notations that do not support rigorous analysis, as well as programming languages that are not fully defined or that defeat automated analysis.
|
From page 43... ...
The marginal cost of supplying different levels of dependability using traditional approaches is depicted by the line labeled "MCTraditional." With perfect competition, the market will reach an equilibrium in which firms supply dependability, DepT0, at the price PT0. Next, consider the introduction of strong software engineering approaches (Figure 1.2)
|
From page 44... ...
and lower price (PS0) .78 It is a new equilibrium because, in a perfectly competitive market, firms that continue to use traditional approaches would be driven out of business by firms using strong approaches.
|
From page 45... ...
(In one incident, avionics software sensed the pilot was performing a touch-and-go maneuver; this was because the wet tarmac did not allow the wheels to turn, so they skidded. The pilot was trying to land but the control assumed otherwise and would not let him deceler 79 Charles Perrow, 1999, Normal Accidents, Princeton University Press, Princeton, N.J.
|
From page 46... ...
All backup tapes of medication orders were corrupted "because of a complex interlocking process related to the database management software that was used by the pharmacy application. Under particular circumstances, tape backups could be incomplete in ways that 80 See Main Commission Accident Investigation -- Poland, 1994, "Report on the accident to Airbus A320-211 aircraft in Warsaw on 14 September 1993." Available online at |
From page 47... ...
Strom, 2005, "Role of computerized physician order entry systems in facilitating medication errors," Journal of the American Medical Association 293(10)
|
From page 48... ...
Feynman, 1986, "Appendix F -- Personal observations on the re liability of the shuttle," In Report of the Presidential Commission on the Space Shuttle Challenger Accident, June. Available online at |
From page 49... ...
NASA's avionics software for the space shuttle, for example, is estimated to have cost roughly $1,000 per line of code (Dennis Jenkins, "Advanced vehicle automation and computers aboard the shuttle." Available online at |
From page 50... ...
Available online at |
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.