Airport Risk Identification and Prioritization Practices (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 This chapter provides a brief summary of the findings from the review of national and international literature to assess existing tools for identifying common hazards found on airports and the processes used for measuring, monitoring, and prioritizing the associ- ated risks. The following topics are covered in this chapter: • A brief history of risk management. • Examples of existing tools for identifying common hazards and other risk management tools available through the FAA and other regulatory agencies. • A brief explanation of enterprise risk management, which is used by many airports through- out the United States. The History of Risk Management Risk management, in the aviation industry, has been used since the Wright brothers. One of Wilbur Wright’s famous quotes is, “The man who wishes to keep at the problem long enough to really learn anything positively must not take dangerous risks. Careless- ness and overconfidence are usually more dangerous than deliberately accepted risks” (McCullough, 2015). Before the 1940s, safety was generally achieved by attempting to control obvious haz- ards in the initial design of the manufactured article. The testing phase of the product would identify unanticipated risks. These newly identified risks would be corrected before manufacturers entered the production phase. In other words, designers used a trial-and- error methodology. In the aviation field this process became known as the fly-fix-fly approach. An aircraft would be designed using the best knowledge available, flown until risks were detected (or it crashed), and then the risks would be mitigated or eliminated. After the aircraft was fixed, it would be flown again. This method worked best with low, slow aircraft. This approach was not acceptable for certain programs—such as nuclear weapons and space travel. The United States Air Force (USAF) and the Department of Defense (DOD) realized that the consequences of accidents were too great. Trial-and-error and fly-fix-fly approaches were not adequate for systems that had to be first-time safe. The cornerstone of risk management included an analysis of likelihood and severity. The transition away from the original trial-and-error approach occurred as both aircraft and weapon systems became more complex and the consequences of accidents became less acceptable (Rodrigues and Cusick, 2011). C H A P T E R 2 Literature Review

6 Airport Risk Identification and Prioritization Practices The 1960s—the U.S. Military and NASA Risk management began to evolve as a separate discipline in the 1960s. The USAF and DOD weapons industries desired a more in-depth, upstream risk management effort. They were designing and deploying the Minuteman intercontinental ballistic missile. A series of Minute- man design–related silo accidents provided at least part of the incentive (Vaughan, 1996). Significant risk management errors were present within civil and military aviation, along with the space program. NASA developed its own risk management program in the 1960s. NASA’s risk management program closely paralleled the USAF approach because the two agencies shared many common contractors, personnel, and missions (Vaughan, 1996). The 1970s and 1980s—Facility Risk Management Throughout the 1970s and 1980s, three factors have driven risk management techniques in areas other than the traditional aerospace, weapons, and nuclear fields. First: the complexity and high cost of many nonflight, nonnuclear projects dictated a more sophisticated upstream risk management approach. Second: product liability litigation pro- vided added incentive to produce safety products. Third: system safety experience began to demonstrate that upstream risk management efforts lead to a better design. Risk management tools and techniques originally considered to be expensive but necessary add-ons proved to be cost-effective planning and review tools (Stolzer, Halford, and Goglia, 2015). The need for a system safety effort for major military construction projects resulted in the development of draft guidelines and facility system safety workshops for the military safety and engineering communities. By the end of the decade, the U.S. Army Corps of Engineers (USACE) facility system safety training programs for government employees were established, and similar courses for contractors were available (U.S Army Corps of Engineers, 2014). The 1990s—Risk-Based Management of Change Before the 1990s, Occupational Safety and Health Administration (OSHA) regulations were almost exclusively compliance based. Very specific rules were written. OSHA inspec- tions also were primarily compliance based. During the 1990s, OSHA began to understand how a change in operations could affect risk management. An analogy to the management of change can be compared to driving a car. If you get in a lane and stay there, you are safe; but when you change lanes, you must be particularly careful and watchful. Much the same can be said about change management created through construction projects (U.S. Army Corps of Engineers, 2014). International Civil Aviation Organization Endorsement of a State-Sponsored Safety Management Program—2008 The United States is a member state of the International Civil Aviation Organization (ICAO) and supported ICAO’s adoption of SMSs as a means of moving aviation to the next level of safety (FAA, 2016). SMS is a formalized process for collecting safety data, identifying hazards and trends, determining safety risk severity, and mitigating risk to an acceptable level. Originally, the FAA’s SMS requirements were aimed at 14 Code of Federal Regulations (CFR) Part 121 air carriers.

Literature Review 7 To promote SMS, ICAO amended Annex 14, Aerodromes, to require states to implement SMS at international airports. The FAA Office of Airports is implementing SMS within its internal organization. In addition, the FAA initiated a rulemaking action to require commer- cial service airports certificated under 14 CFR Part 139 to implement SMS. Starting in 2008, numerous airport operators volunteered to participate in SMS pilot studies. The majority of the airports that participated received federal financial assistance through the Airport Improvement Program. These airport operators experienced the challenges and benefits of developing and implementing SMS at 14 CFR 139 airports. In spring 2011, as part of the rulemaking effort, the FAA formed a team to gather infor- mation from the then-ongoing pilot studies and prepared a technical report documenting the participating airports’ experiences and lessons learned (Maurino, 2007). Internal and External SMS Efforts in the FAA Airports Organization The FAA is continuously developing and implementing SMS for airports both internally and externally. The FAA Airports organization has documented these internal SMS efforts as well as external SMS efforts. The Internal SMS Efforts website (see https://www.faa.gov/ airports/airport_safety/safety_management_systems/internal) provides advisory guidance and policies applicable to airports that aid in the internal efforts of incorporating safety risk management into the airports’ planning and development processes. The External SMS Efforts website (see https://www.faa.gov/airports/airport_safety/safety_management_ systems/external) also lists several related supporting documents about external SMS efforts. The FAA external SMS efforts focus on how the FAA will incorporate the SMS requirement of ICAO Annex 14, Volume 1 (Aerodrome Design and Operations) into the more than 540 U.S airports certificated under Part 139. Part 139 Rulemaking and Airport SMS Pilot Studies Under multiple Airport SMS pilot studies, several airports received Airport Improve- ment Program grants to develop their SMS manual and their implementation plan. Airports that participated in the study were required to follow a Statement of Work and Pilot Study Participant Guide, which detailed the deliverables and time frames for the study. A second FAA SMS pilot study gathered information on scalability and how smaller airports might implement SMS. Risk Management Observations from the Pilot Studies The FAA website lists key features related to risk management (see http://www.regulations. gov and search for docket number FAA-2010-0997 for a series of useful documents). The key features include • A description of risk management processes, including application of the “five phases of safety risk management,” as discussed in the Advisory Circular 150/5200-37; • Guidance on the use of safety risk management (SRM) and trend analysis; • A defined process for documenting the results of SRM, including a description of how documents will be stored—i.e., electronic or paper; and • Descriptions of how top management will follow up on SRM to ensure safety mitigation strategies are appropriate.

8 Airport Risk Identification and Prioritization Practices The FAA’s Advisory Circular 120-92B indicates that risk management is a fundamental component of SMS. The principal steps in the SRM process include identification of a poten- tial hazard, analysis of the risk, evaluation of the risk, and development of an action plan to mitigate the risk if necessary. The Advisory Circular states that the hazard identification stage should consider all the pos- sible sources of system failures (i.e., operations, equipment, people, and procedures). Possible sources of system failure could include • Equipment (example: construction equipment on a movement surface); • The operating environment (example: cold, night, low visibility); • The human element (example: shift work); • Operational procedures (example: staffing levels); • Maintenance procedures (example: nightly movement area inspections by airport electri- cians); and • External services [example: ramp traffic by fixed-base operators (FBO) or law enforcement vehicles]. Although airports informally consider many elements of risk management in making man- agement decisions, many currently do not have a formal risk management process for aviation safety. Results found that many of the airports were willing to share lessons learned and the documentation they developed during the pilot studies. The FAA posted to the SMS docket tools used by the participants, including SMS gap analysis, SMS manuals, and SMS implemen- tation plans (see http://www.regulations.gov and search docket number FAA-2010-0997). Pilot study airports noted the following results: Workload impact: Pilot study airports found the workload impact of SMS-related elements manageable. Although unable to make staffing changes, many airports were still able to reason- ably accommodate development of SMS guidance and implementation. Gap analysis: The airport operators found the gap analysis useful. It enabled airport operators to identify those requirements of SMS that are not part of 14 CFR 139 compliance activities and showed that many 14 CFR 139 activities (such as daily self-inspection, airport emergency plans, and notifications) can serve as a foundation for the components and elements of SMS. Benefits: Overall, airport operators benefited from improved communication and increased safety awareness. Guidance: In general, airport operators found the guidance was sufficient. They sug gested that the FAA further clarify areas such as SMS development, support tools, and templates. Risk Tools Used by the Air Traffic Organization There are multiple SMS processes followed in the FAA. Airports may be able to use or adapt tools provided in Air Traffic Organization (ATO) SMS documents. The latest SMS Manual for the ATO emphasizes the practicality of safety risk management. The manual states the importance of a practical use of SRM as a formal safety and risk assessment process. Specifically, its philosophy is easily understood outside of the technical realm of aviation. For example, a person performs SRM each time he or she crosses the street. The individual identifies hazards (cars passing), analyzes and assesses the risk (potential to be struck and severity if he or she is), and explores ways to reduce the perceived risk (looking both ways for traffic and/or heeding pedestrian signals) to an acceptable level before proceeding. It is necessary to make the approach to managing safety risk into a formalized, objective process. This helps ensure the effective management and reduction of a risk; therefore, SRM provides a means to

Literature Review 9 • Identify potential hazards and analyze and assess safety risk in ATO operations and National Airspace System (NAS) equipment; • Define safety requirements to reduce risk to an acceptable level; • Identify safety performance targets, the measurable goals used to verify the predicted residual risk of a hazard; and • Create a plan that an organization can use to determine whether expected risk levels are met and maintained. Risk Management Analysis Phases Risk analysis is broken down into five phases (see Figure 2). Consistent with ICAO guide- lines and best practices, these five phases apply to all risk management activity, whether the activity pertains to airports, air traffic organization operations, maintenance procedures, or equipment development. Systematically completing the steps outlined in the five phases supports a thorough and consistent risk analysis. The Preliminary Hazard Analysis Form The FAA Office of Airports, Safety Management System (SMS) Desk Reference, Version 1.0, uses a preliminary hazard analysis (PHA) form to organize the process of managing risk. The PHA serves as a guide for the hazard and risk analysis (Figure 3). The Preliminary Hazard Analysis form begins by identifying a combination of hazards, causes, effects, and system states. The next step is to assign each item to its appropriate category Figure 2. Safety risk management safety assessment process. (FAA. AC 150/5200-37A, Safety Management Systems for Airports. 2016.)

10 Airport Risk Identification and Prioritization Practices Figure 3. Sample preliminary hazard analysis worksheet (FAA Office of Airports 2012, Appendix I). (i.e., hazard, cause, effect, or system state). The resulting hazards, causes, effects, and system states can then be worked into the appropriate hazard analysis tool. The term “system state” is an expression of the various conditions, characterized by quantities or qualities, in which a system can exist. For airport-related risk analysis, this may include instru- ment versus visual meteorological conditions, snow, ice, or rain events versus normal conditions or operations during construction. For example, a hazard’s cause or effect may differ depending on whether it occurs in such varied circumstances as • Instrument or visual meteorological conditions • Day or night operations • Taxiways/runways contaminated with snow or dry taxiways/runways • Construction operations or normal operations without construction activities • Airshow operations or return to normal activities at the conclusion of the airshow • Changes in wildlife activity Other Accepted Tools and Techniques If the risk analysis calls for an additional means to identify hazards and compare solutions, users can select the methodology that is most appropriate for the type of system being evaluated. When selecting hazard identification/analysis tools, it is important to consider • The necessary information and its availability; • The timeliness of the necessary information; • The amount of time required to conduct the analysis; and

Literature Review 11 • The tool that will provide the appropriate systematic approach for – Identifying the greatest number of relevant hazards, – Identifying the causes of the hazards, – Predicting the effects associated with the hazards, and – Assisting in identifying and recommending risk mitigation strategies. Table 1 summarizes several hazard identification tools available. Tool or Technique Summary Description Bowtie Model Risk Assessment The Bowtie Model Risk Assessment is used to analyze and communicate risk scenarios. A Bowtie diagram gives a visual summary of all plausible incident scenarios that could exist around a certain hazard. The Bowtie also represents what an organization does to control those scenarios by identifying safety barriers. Comparative Safety Assessment The Comparative Safety Assessment provides management with a list of all of the hazards associated with a National Airspace System change, along with a risk assessment for each alternative hazard combination that is considered. It is used to rank the options for decision-making purposes. The Comparative Safety Assessment’s broad scope makes it an excellent way to identify issues that may require more detailed hazard identification tools. Failure Mode and Effect Analysis The Failure Mode and Effect Analysis determines the results or effects of sub-element failures on a system operation and classifies each potential failure according to its severity. Failure Modes, Effects, and Criticality Analysis The Failure Modes, Effects, and Criticality Analysis is an essential function in design from conception through development. The analysis is iterative to correspond with the nature of the design process itself. It identifies component and subsystem failure modes (including the effect of human error), evaluates the results of the failure modes, determines rates and probability, and demonstrates compliance with safety requirements. Fault Hazard Analysis The Fault Hazard Analysis is a deductive method of analysis that can be used exclusively as a qualitative analysis or, if desired, can expand to a quantitative one. The analysis requires a detailed investigation of subsystems to determine component hazard modes, causes of these hazards, and resultant effects on the subsystem and its operation. Fault Tree Analysis A Fault Tree Analysis is a graphical design technique that can provide an alternative to block diagrams. It is a top-down, deductive approach structured in terms of events. It is used to model faults in terms of failures, anomalies, malfunctions, and human errors. Job Task Analysis The foundation of the performance of a Human Error Analysis is a Job Task Analysis, which describes each human task and subtask within a system in terms of the perceptual (information intake), cognitive (information processing and decision-making), and manual (motor) behaviors required of an operator, maintainer, or support person. The Job Task Analysis should also identify the skills and information required to complete tasks; equipment requirements; the task setting, time, and accuracy requirements; and the probable human errors and consequences relating to these areas. There are several tools and techniques for performing task analyses, depending on the level of analysis needed. Table 1. Hazard identification tools. (continued on next page)

12 Airport Risk Identification and Prioritization Practices Tool or Technique Summary Description Subsystem Hazard Analysis In acquisitions, the general purpose of the Subsystem Hazard Analysis is to perform a safety risk assessment of a system’s subsystems and components at a more detailed level than that provided in a Preliminary Hazard Analysis. System Hazard Analysis In acquisitions, the general purpose of the System Hazard Analysis is to perform a detailed safety risk assessment of a system, particularly the interfaces of that system with other systems and the interfaces between the subsystems that compose the system under study. The System Hazard Analysis and Subsystem Hazard Analysis are interrelated analyses that may be done concurrently. What-If Analysis The What-If Analysis methodology identifies hazards, hazardous situations, or specific accident events that could produce an undesirable consequence. One can use the What-If Analysis as a brainstorming method. Source: Air Traffic Organization, Safety Management System Manual, April 2019. Operational Hazard Assessment The Operational Hazard Assessment is a development tool based on the assessment of hazard severity. It establishes how safety requirements are to be allocated between air and ground components and how performance and interoperability requirements might be influenced. Scenario Analysis The Scenario Analysis tool identifies and corrects potentially hazardous situations by postulating accident scenarios in cases in which it is credible and physically logical to do so. Table 1. (Continued). Levels of Risk The FAA has identified several risk tables when managing risk within its own organizations. Risk levels are typically organized in a matrix (see Figure 4) and defined as • High risk—Unacceptable level of risk. The proposal cannot be implemented or the activity continued unless hazards are further mitigated so that risk is reduced to medium or low level. Tracking and management involvement are required, and man- agement must approve any proposed mitigating controls. Catastrophic hazards that are caused by – Single-point events or failures, – Common-cause events or failures, or – Undetectable latent events in combination with single-point or common-cause events are considered high risk, even if extremely remote. • Medium risk—Acceptable level of risk. Minimum acceptable safety objective; the pro- posal may be implemented or the activity can continue, but tracking and management are required. • Low risk—Target level of risk. Acceptable without restriction or limitation; the identified hazards are not required to be actively managed but are documented. MITRE Risk Matrix There are numerous risk matrix charts used throughout the world. Another example of a more detailed risk matrix came from MITRE Corporation in 2009 (see Figure 5). It identified several new considerations for both severity and likelihood.

Literature Review 13 Risk Management and FAA Order 8900.1 FAA Order 8900.1, Volume 17, ties risk management into a cycle of continuous moni- toring and improvement. The risk management and safety assurance components are closely linked. Risk management functions ensure that hazards and their associated risks are identified, analyzed, and assessed, and that mitigations are put in place when necessary. Safety assurance processes then take over, using data to evaluate whether the mitigations are having the desired effect. Figure 6 depicts the risk management/safety assurance relationship. Figure 4. Risk matrix—commercial operations/large transport category and general aviation operations/small aircraft and rotorcraft (FAA Order 8040.4B, 2017).

14 Airport Risk Identification and Prioritization Practices Figure 5. Risk assessment matrix (MITRE, www.mitrecaasd.org/SMS/doc/Sample_Risk_Matrix5.pdf). Risk Management and FAA Advisory Circulars One step in the ICAO risk management process is to identify existing controls to manage risks (Maurino, 2007). The FAA has published numerous advisory circulars (see Table 2) to help airports manage risks. These advisory circulars, found at www.faa.gov, give detailed information on risk identification and successful strategies to control risk. Enterprise Risk Management ERM is a holistic, interdisciplinary, enterprise-wide approach to identifying, priori- tizing, and treating an organization’s entire portfolio of risks. Though broad, the goal is not to identify the entire universe of risks, but rather to focus on risks to the organization’s strategic goals, purpose, and mission. ERM was established as a structured discipline in 2004 with the introduction of the Committee of Sponsoring Organizations’ Enterprise Risk Management framework. The International Organization of Standardization (ISO) issued its ERM Stan- dard (ISO 31000) in 2009. Both provide guidance on expanding the understanding of risk beyond traditional hazard or insurable risk, and they provide focus on the important role that an effective risk management process has in protecting and creating value and supporting risk-informed decision-making at all levels of an organization.

Literature Review 15 SRM SA Risk Assessment System Assessment System Description (Analysis) System Monitoring Data Acquisition Hazard Identification Risk Analysis Analysis of Data Risk Control Corrective Action Description and Context Specific Information Analysis Assessment Action: Problem Resolution Decision Steps Figure 6. Risk management/safety assurance relationship (FAA Order 8900.1 CHG 374, 2015). ERM processes use the strategy of coordinated risk management that places an emphasis on cooperation and coordination among departments to manage the full range of an organization’s risks. ERM offers a framework to more effectively manage uncertainty, preventing or responding to adverse risks as needed and maximizing opportunities as they arise. Risk as Opportunity in Enterprise Risk Management In the past, organizations took a defensive posture towards risks. Risks were hazards that should be avoided. ERM recognized that risks may open the door to opportunities. An orga- nization may have a special ability to manage a risk where other competing organizations do not. These unique risk management capabilities give an organization a special ability to optimize those risks (Miccolis, 2000). Risk Assessment and Prioritization in Enterprise Risk Management Enterprise risk management uses risk strategies on the basis of likelihood and severity. This is similar to the other accepted methods of risk management. ERM will go beyond the typical risk matrix. An airport using ERM principles would prioritize a number of risks, including its workers’ compensation exposure (hazard), potential legal action risks

16 Airport Risk Identification and Prioritization Practices Advisory Circular Number Title 150/5200-36B Qualifications for Wildlife Biologist Conducting Wildlife Hazard Assessments and Training Curriculums for Airport Personnel Involved in Controlling Wildlife Hazards on Airports 150/5370-2G Operational Safety on Airports During Construction 150/5200-33B Hazardous Wildlife Attractants on or Near Airports 150/5200-38 Protocol for the Conduct and Review of Wildlife Hazard Site Visits, Wildlife Hazard Assessments, and Wildlife Hazard Management Plans 150/5210-24 Airport Foreign Object Debris (FOD) Management 150/5340-1L Standards for Airport Markings 150/5220-25 Airport Avian Radar Systems 150/5210-5D Painting, Marking, and Lighting of Vehicles Used on an Airport 150/5200-30D Airport Field Condition Assessments and Winter Operations Safety 150/5220-22B Engineered Materials Arresting Systems (EMAS) for Aircraft Overruns 150/5200-31C Airport Emergency Plan (Consolidated AC Includes Change 2) 150/5200-34A Construction or Establishment of Landfills Near Public Airports 150/5200-18C Airport Safety Self-Inspection 150/5340-30J Design and Installation Details for Airport Visual Aids 150/5200-28F Notices to Airmen (NOTAMs) for Airport Operators 150/5320-6F Airport Pavement Design and Evaluation 150/5345-44K Specification for Runway and Taxiway Signs 150/5210-20A Ground Vehicle Operations to Include Taxiing or Towing an Aircraft on Airports 150/5210-17C Programs for Training of Aircraft Rescue and Firefighting Personnel 150/5220-20A Airport Snow and Ice Control Equipment 150/5340-26C Maintenance of Airport Visual Aid Facilities 150/5345-27E Specification for Wind Cone Assemblies 150/5340-5D Segmented Circle Airport Marker System 150/5230-4B Aircraft Fuel Storage, Handling, Training, and Dispensing on Airports 150/5220-26 Airport Ground Vehicle Automatic Dependent Surveillance—Broadcast (ADS-B) Out Squitter Equipment 150/5300-17C Standards for Using Remote Sensing Technologies in Airport Surveys (Consolidated to Include Change 1) 150/5345-28G Precision Approach Path Indicator (PAPI) Systems 150/5220-10E Guide Specification for Aircraft Rescue and Fire Fighting (ARFF) Vehicles 150/5370-17 Airside Use of Heated Pavement Systems 150/5210-23 ARFF Vehicle and High Reach Extendable Turret (HRET) Operation, Training and Qualifications 150/5220-17B Aircraft Rescue and Fire Fighting (ARFF) Training Facilities 150/5210-13C Airport Water Rescue Plans and Equipment 150/5345-12F Specification for Airport and Heliport Beacons 150/5340-18F Standards for Airport Sign Systems 150/5300-18B General Guidance and Specifications for Submission of Aeronautical Surveys to NGS: Field Data Collection and Geographic Information System (GIS) Standards 150/5210-14B Aircraft Rescue Fire Fighting Equipment, Tools and Clothing 150/5210-18A Systems for Interactive Training of Airport Personnel 150/5210-15A Aircraft Rescue and Firefighting Station Building Design 150/5210-7D Aircraft Rescue and Fire Fighting Communications 150/5300-16A General Guidance and Specifications for Aeronautical Surveys: Establishment of Geodetic Control and Submission to the National Geodetic Survey 150/5200-29A Announcement of Availability of Airport Self-Inspection DVD 150/5210-6D Aircraft Fire Extinguishing Agents 150/5210-22 Airport Certification Manual (ACM) 150/5345-43J Specification for Obstruction Lighting Equipment Table 2. Advisory circulars to help airports manage risks.

Literature Review 17 (financial), labor relations risk (operational), and infrastructure obsolescence risk (strategic) (Berry and Phillips, 1998). Unlike previous risk management practices, the concept of ERM embodies the notion that risk analysis cuts across the entire organization. The goal of ERM is to better understand both risks and opportunities to better manage enterprise risk exposure to the level desired by senior management. Enterprise risk management offers a number of benefits (Sanderson and Koritzinsky, 1999): • Aligns risk appetite and corporate strategy • Links business growth, risk, and returns • Improves risk responses • Reduces operational surprises and losses • Manages enterprise-wide risks • Recognizes and acts upon opportunities • Deploys resources effectively Organizations have traditionally used a “silo” approach to risk management that looks at the individual performance of a business unit instead of taking a more holistic approach that looks at the long-term impact on risk and capital needs of the entire enterprise. Organizations, especially those in insurance, have long identified and quantified risks, though this has commonly been done in a vacuum that separates risk management, capi- tal management, and financial management. Therefore, in order to effectively allocate resources, satisfy all stakeholder demands, manage the diverse risks, allocate capital to those areas that add value, and measure the organization’s performance, it is critical that the “silos” be permanently linked. Summary Risk management in the aviation industry has been used since the Wright brothers. One of their famous quotes, by Wilbur Wright, is “The man who wishes to keep at the problem long enough to really learn anything positively must not take dangerous risks. Carelessness and over- confidence are usually more dangerous than deliberately accepted risks” (McCullough, 2015). Making risk management work depends on process but more importantly on people with knowledge and experience in the disciplines relevant to the product and with the resolve to identify and address the risks that could influence program objectives. The authors of this study searched the term “risk assessment tools” in Google. The search yielded 652,000,000 hits: the number of risk assessment tools and processes is extensive. In the end, what matters most is the quality and effectiveness of the program’s risk mitigation plans and their implementation in reducing the risks to realizing program objectives, not the process itself. The steps of the risk management process are generally applicable to the man- agement of risks and are used in multiple phases of the risk management life cycle. At the same time, the nature of the specific actions taken for each step typically will be different, depending on the individual taking them. The differences in the specific actions are driven by the chang- ing types of risk, the information and tools available, the outcomes that need to be achieved, the degree of maturity and stability that must be demonstrated, and the residual risks that are tolerable following mitigation efforts.