Applying Risk Analysis, Value Engineering, and Other Innovative Solutions for Project Delivery (2017)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

66 This report includes the research results related to RA performed. This includes the following: • Risk Analysis Literature Review • Risk Analysis Surveys • Summary and Conclusions Risk Analysis Literature Review The research team reviewed 27 documents related to RA as a part of its initial research effort (see Table 6). This includes the applicable policy, procedure, and guidance documents from 14 state DOTs. In addition, a number of technical papers on RA published by the FTA, the FHWA, and TRB were also reviewed. The findings from this review point to the following general conclusions: • RA activities are an emerging effort that a majority of state DOTs are undertaking; however, many agencies are in the initial stages. Most agencies that perform RA activities tend to per- form less sophisticated qualitative risk analyses (i.e., rating of probabilities and impacts from a calibrated scale of very low to very high in a matrix format). • There is a high degree of inconsistency in terms of terminology. The terms Risk Analysis, Risk Assessment, and Risk Management are used interchangeably in several of the docu- ments reviewed. Risk Analysis and Risk Assessment may be used as terms for performing risk identification and risk analysis or for conducting Risk Management efforts that include risk response planning and risk monitoring and control. • There are a limited number of state DOTs that have a high degree of sophistication in undertak- ing RA. Agencies that can be considered more advanced perform Risk Management activities (risk identification, RA, risk response planning, and risk monitoring and control) throughout the entire project delivery life cycle in an iterative fashion at various scheduled milestones. In addition, these agencies tend to perform more detailed quantitative risk analyses involving the use of Monte Carlo simulation techniques. • A significant number of state DOTs perform RA; however, there are significant inconsisten- cies relative to how the assessments are conducted (i.e., qualitative versus quantitative), the intent and focus of the risk assessments, when they are conducted during the project life cycle, extent of the analysis performed, and who performs the risk assessment (i.e., internal versus consultant staff). • Many states have developed certain guidelines for conducting RA efforts; however, the consis- tency of information and thoroughness of the guidelines varies greatly. Some states only iden- tify the necessity to conduct a RA while others prescribe detailed Risk Management activities. C h a p t e r 3 Risk Analysis Research Results

risk analysis research results 67 No. State/Publisher Title of the Literature/Report Date 1. California Project Risk Management Handbook June 2012 2. Florida Project Management Handbook: Chapter 19 Project Risk Management March 4, 2008 3. Minnesota Risk-Based Engineers Estimate March 2015 4. Montana Risk Management Guideline: Managing project costs through identification and management of risks DRAFT January 2014 5. Nevada Risk Management and Risk-Based Cost Estimation Guidelines August 2012 6. New Jersey Risk Management Guideline January 2015 7. New York Risk Management for Project Development April 9, 2009 8. Texas Risk-Based Construction Cost Estimating Reference Guide 2014 9. Virginia Project Management Procedure: Project Risk Management February 1, 2015 10. Washington Project Risk Management Guide November, 2014 11. TRB NCHRP Report 658: Guidebook on Risk Analysis Tools and Management Practices to Control Transportation Project Costs 2010 12. TRB NCHRP Web-Only Document 183: Guidance for Managing NEPA-Related and Other Risks in Project Delivery Volume 1 October 1, 2011 13. TRB NCHRP Web-Only Document 183: Guidance for Managing NEPA-Related and Other Risks in Project Delivery Volume 2 March 1, 2014 14. TRB ACRP Report 76: Addressing Uncertainty about Future Airport Activity Levels in Airport Decision Making 2012 15. TRB NCHRP Report 706: Uses of Risk Management and Data Management to Support Target-Setting for Performance- Based Resource Allocation by Transportation Agencies 2011 16. North Carolina Development of Funding Project Risk Management Tools November 5, 2013 17. Utah UDOT Project Manager Guide Chapter 7: Managing Project Risk 2013 18. FHWA Guide to Risk Assessment and Allocation for Highway Construction Management October 2006 19. Montana Highway Project Cost Estimating and Management February 2009 20. TRB NCHRP 20-24: Executive Strategies for Risk Management by State Departments of Transportation May 2011 21. FTA Risk Analysis Methodologies and Procedures June 2004 22. FHWA Risk Assessment for Public-Private Partnerships: A Primer December 2012 23. FHWA Guidebook for Risk Assessment in Public Private Partnerships December 2013 24. FHWA P3-Value: Risk Assessment Tool User Manual April 19, 2013 25. TRB Guide for the Process of Managing Risk on Rapid Renewal 2014 Projects (SHRP 2 Report S2-R09-RW-2) 26. 27. Missouri DOT Project Delivery Method Determination and Risk Assessment (Category 149, Engineering Policy Guide) March 28, 2014 Ontario Ministry of Transportation 1. Guidelines for the Use of Cost Risk Assessments 2. CRA Fact Sheet August 17, 2015 Table 6. RA documents reviewed.

68 applying risk analysis, Value engineering, and Other Innovative Solutions for project Delivery • RA efforts are often conducted as an integrated effort of project management. Many DOTs that have formally embraced RA have issued project management delivery directives that mandate the use of such efforts at specific project milestones. Some states have formal poli- cies about which projects are required to perform RA; however, within such policies there are many references to discretionary application of RAs on an as-needed basis as determined by the project teams on large, complex, or issue-ridden projects. It should be noted that very little specific information is publicly available about when a discretionary RA should be performed. • The use of either internal or consultant staff for RAs is dependent on the organizational expertise in terms of personnel capabilities. Many states are developing internal personnel, but most do not have formal programs staffed with significantly experienced internal personnel managing RA programs such as what is done for VE. As a result, many states tend to rely on consul- tant support for RA efforts and engage in actions for management of risks through project managers and project engineers that are assigned project delivery responsibilities. Consultants may coordinate with DOT staff to plan and facilitate workshops, conduct risk analyses, and develop risk response strategies and action plans for project managers. Then project managers presumably carry forward the actions necessary to manage the risk. Very few DOTs have the internal resources, experience, or staff to perform all actions of the RA process proficiently. As a result, conducting RA efforts tends to involve consultant support. Note that this relationship is directly correlated to the level of maturity of an organization’s RA program in that more mature organizations tend to rely less on consultant support, but may still use consultants in their RA efforts for complex or challenging projects. • Very few states have a specific system in place to track the efficacy of their RA efforts or the performance of their RA program. States with formal RA policies track whether a RA was conducted or not, but there is little information being collected regarding the implementa- tion of risk response strategies, reduction in change orders, or cost and time savings resulting from the RAs. All states with RA policies do reference the use of risk registers for capture and organization of risk data and information. • Some specific connections or references are made between the state’s RA program and other design and project review processes such as VE; however, little or no information was avail- able regarding the linkage of RA to CRs. The more sophisticated states engaging in RA efforts also tend to perform some level of combined RA/VE efforts that use RA to identify issues and VE to respond to issues and develop alternative solutions. Risk Analysis Surveys A formal RA Survey was distributed to representatives performing RA activities at various state DOTs on August 17, 2015. The survey was sent to individuals in various state DOTs as well as state FHWA liaisons and individuals from other transportation agencies. Thirty surveys were sent out based on the initial request for contacts distributed to state DOTs and on requests for information about RA staff from the VE survey. The research team received 14 responses—a response rate of 43%. The survey on transportation agency practices in RA indicated that use of these types of tech- niques was more prevalent than CRs to support project delivery. Further, survey results indicated that use of RA tools and techniques is becoming more commonplace in the delivery of transporta- tion projects. Most agencies do not have an established dollar threshold for performing of RA and commonly include it as part of project delivery activities. Most organizations that do perform RA indicated that they are conducting qualitative types of analysis. A very small number of survey respondents indicated more sophisticated levels of quantitative modeling; however, this is typically only done on projects that are $100 million or more (in capital cost). Most respondents indicated that agencies will typically conduct a single

risk analysis research results 69 RA. A smaller percentage performed continuous RA activities throughout the project delivery life cycle. It is noted that most organizations do not spend their RA efforts developing risk response strategies and action plans or in engaging in risk monitoring and control. RA efforts are most commonly performed as a combination of internal and external consultant staff. RA was noted to be done in conjunction with VE 42% of the time and determined on a case-by-case basis for projects. The following is a summary of survey findings: • Approximately 50% of respondents indicated Design was the Division/Office/Bureau respon- sible for RA. The remaining 50% indicated they had Project Management Offices or specific RA units within their organizations that were responsible for RA. • Approximately 70% of responding transportation agencies indicated that they had a formal RA policy. – Those agencies with formal policies indicated these policies were created recently, that is, 2010 and later. • Approximately 70% of responding organizations indicated that specific individuals are delegated to manage risks for projects. • Approximately 70% of responding transportation agencies indicated that they did not have an established dollar threshold for performing RA activities; however, the remaining 30% either did have a threshold (commonly $10 million or more for RA) or had RA mandated on all projects regardless of dollar amount. Some respondents indicated that they only do RA on projects of $500 million or more, as mandated by FHWA. • The responding transportation agencies indicated that the most common use of RA was for highway and bridge projects. Other uses included evaluation of business processes, evaluation of construction claims, or applications of enterprise RA. • Generally, within transportation organizations, roughly 62% of respondents indicated that opin- ions are divided as to the positives and negatives of the use of RA in project delivery. Another 23% indicated that most project managers see the benefits and believe it has a positive impact on project delivery. The remaining 15% of respondents indicated that RA is an activity to be avoided by proj- ect managers and it is generally seen as a “check the box” requirement if mandated for a project. • Almost all survey respondents strongly agreed that RA is a performance and/or quality improvement method. It can aid in controlling costs, it helps to control schedule, and can help to reduce project uncertainties leading to unforeseen scope, schedule, or cost issues. • All respondents indicated that RA is conducted during environmental studies or preliminary design. The usage declines the further along agencies are in the project delivery process: 75% of respondents use RA in planning and programming, 67% use RA during final design, and 42% use RA during construction. • The various disciplines engaged when conducting a RA are based on the project type and specific challenges being faced when risk workshops are conducted. • Approximately 70% of responding organizations internally conduct RA activities. Another 23% of respondents rely on internal staff and external consultants and only 7% of organiza- tions rely solely on external consultants. • When conducting risk workshops, team members are drawn from within the organization 42% of the time, whereas 50% of the time workshop teams consist of internal staff and external consultant staff. • When RA activities are conducted for a project they are conducted multiple times throughout the project life cycle 58% of the time. The remaining 42% of the time they are conducted a single time. • When risk workshops are conducted the modeling of risk is done during the workshop using qualitative means 23% of the time. The remaining 77% of the time risk models are prepared outside of the risk workshop in quantitative form.

70 applying risk analysis, Value engineering, and Other Innovative Solutions for project Delivery • RA workshops are commonly conducted as a single workshop session for a project; however, 40% of agencies are exploring options to conduct workshops at various intervals during project delivery. • The vast majority of organizations conducting RA perform qualitative analysis. Some organi- zations perform semi-quantitative analysis, while a smaller number of organizations perform advanced quantitative modeling (including Monte Carlo simulation). • Approximately 46% of organizations indicated they use a specific threshold for requiring quantitative RA. The most commonly cited threshold was $100 million or more in capital costs for a given project. • For those transportation agencies that engage in RA, 58% of them also engage in risk response planning. The other 42% of organizations do not perform formal risk response planning. • Approximately 50% of respondents indicated that less than 10% of the total time expended on Risk Management is expended on risk response planning and risk monitoring and control. One-third of respondents indicated that 30 to 60% of the time expended on Risk Management is expended on risk response planning and risk monitoring and control. • All respondents indicated that their agency has their own internal RA spreadsheets developed in MS Excel. Another 42% of respondents indicated use of publicly available Risk Manage- ment software tools. In addition, 17% of respondents noted use of Palisade @Risk for con- ducting Monte Carlo simulations. • Only 23% of respondents indicated that additional software beyond Risk Management tools was used during RA workshops. Additional software tools include the following: – Roadway design software – Project scheduling software – Traffic modeling software – Project cost estimating software • None of the responding agencies indicated the use of a clause in construction contract docu- ments requiring RA during construction. • Approximately 46% of respondents indicated that a feedback loop between Design, Construc- tion, and Project Management was used in the tracking of project risks. • It was noted that 8% of respondents indicated that RA was always performed in conjunction with VE. Another 42% of respondents indicated that RA is sometimes performed with VE or that its use is determined on a case-by-case basis. • Respondents generally indicated that RA is rarely done in conjunction with CRs. • Responding transportation agencies who conduct RA activities provided the following percentages for training: – 0% indicated they always provide training – 39% indicated they sometimes provide training – 23% indicated they never provide training – 38% indicated they provide training on a case-by-case basis Summary and Conclusions In summary, RA efforts are becoming more commonplace in the delivery of projects. Many state DOTs have either implemented or are working to develop internal policies and proce- dures for the analysis and management of risk. A limited number of state DOTs and trans- portation agencies can be considered very sophisticated in their approaches, processes, and techniques for RA; however, many states do perform some level of RA and capture of risks in a register. It can be concluded that there is a general recognition across agencies that the man- agement of uncertainty and risk can be an effective project delivery tool for control of scope, schedule, and budget.

risk analysis research results 71 In light of the strengthening emergence of RA within state DOTs and transportation agencies, the most effective organizations have standardized processes and tools. The research team sug- gests that transportation agencies employ a standard process-driven approach to analyzing and managing risk. This can be most accomplished via the use of an MS Excel workbook tool—all state DOTs indicated use of the Microsoft Office suite. The MS Excel workbook ideally should be uniformly applied to projects and walk users through the process of Risk Identification, RA (qualitative and/or quantitative), Risk Response Planning, and Risk Monitoring and Control while capturing and organizing all of the necessary information and data. Applying this work- book to projects would also allow for continuous editing and updating of risk information and data throughout the project delivery life cycle and provide a record of the efforts taken to manage risk. Further, the archive of risk data and information for any given project should also be reviewed and transferred into a lessons learned or feedback process for use by other project teams. RA varied in terms of the type of methods used to conduct the analysis. Analysis types ranged from qualitative to quantitative employing various methods to characterize the degree of risk exposure to project costs, durations, and scope. For RA, qualitative analysis is effective on smaller and less complex projects, and increasing degrees of quantitative analysis (i.e., quanti- fication of risks in the register, Monte Carlo simulations, and integrated cost and schedule risk models) are effective on larger and more complex projects. In general, qualitative analysis is recommended for all projects at all stages of the project delivery life cycle; however, quantita- tive analysis practices are more effective on larger projects. Based on the information obtained during this project and from effectively functioning DOTs, it can be concluded that quan- tification of risks in the register is recommended for projects greater than $50 million, and more complex simulation models (Monte Carlo methods and integrated cost and schedule risk modeling) should be considered for projects greater than $100 million. It was found that there are some existing risk modeling and simulation tools available (e.g., Washington State DOT and Utah DOT RA tools) that can be applied to projects that are generally effective for analysis; however, for more complex and larger projects and programs, the research team recommends using Monte Carlo methods in combination with integrated cost and schedule risk modeling approaches, as demonstrated in the research by the most proficient and effectively functioning DOTs performing RA. The most important part of an effective RA process is to plan for and respond to the uncer- tainties and risks that have been identified and prioritized by the organization. It is important to explore a complete identification and evaluation of risk response strategies and actions plans for all plausible scenarios. Effectively functioning RA programs at state DOTs commonly leverage VE as a tool in Risk Response Planning. VE’s structured process and techniques in brainstorm- ing, evaluation, and development of concepts provides more rigidity into the Risk Management process by exploring and evaluating multiple approaches to managing each risk. Integration of VE and RA processes and approaches results in a more robust and effective project delivery management tool to control scope, schedule, and budget. Based on the findings of the research, it can be concluded that VE is an effective “engine” for Risk Response Planning by providing a structured approach and method to fully explore the means and methods to minimize threats and maximize opportunities throughout the project delivery life cycle. The integration of RA and VE into a cohesive process is not uncommon and is found to be one of the most effective practices available. In terms of integrating effective practices and innovative methods for project delivery, the integration of RA and VE has been proven; however, CRs have generally not been integrated with RA efforts. The concept of “constructability” is inherently one that deals with risk related to the construction of a project. As a result, efforts to further integrate CRs with RA efforts should be undertaken by all state DOTs and transportation agencies. There is the synergy of

72 applying risk analysis, Value engineering, and Other Innovative Solutions for project Delivery the RA process providing a structured approach to conduct the analysis and quantify poten- tial construction-related threats and opportunities manifested in staging, phasing, productivity, traffic management, biddability, buildability, and so forth. Further, integration of RA with CR efforts provides a framework for continuous qualification and quantification of the associated construction phase elements throughout project delivery. In conclusion, the research team pursued the following activities in developing a RA tool: • Establish a process driven template (i.e., incorporates Risk Identification, RA, Risk Response Planning, and Risk Monitoring and Control) for RA in MS Excel for use by state DOTs and transportation agencies to manage risk and uncertainties continuously throughout the project delivery life cycle. • Define parameters for the application of qualitative versus quantitative methods of RA in terms of project size and scale (i.e., capital costs and complexity). – Use qualitative and quantitative tools that are publicly available. – Further develop tools for use by DOTs and transportation agencies. • Develop an approach to integrate CRs with RA and VE. – Establish VE as the ideal mechanism for performing Risk Response Planning. – Integrate RA with CRs to capitalize on their commonalities to target outcomes of each respective analysis type and eliminate redundancy in activities.