Estimating Highway Preconstruction Services Costs - Volume 2: Research Report (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Introduction 1.1 Background The issue of accurate estimating of preconstruction services (PCS) costs is essentially tied to the efficient use of available public capital (Janacek 2006). Early estimates conducted dur- ing the planning phase often become legislative authorizations and turn into project budgets before the final scope of project work is adequately quantified (Anderson et al. 2007). Addi- tionally, since preconstruction costs are by definition a small portion of the total project delivery cost, they are typically esti- mated as a standard percentage of estimated construction costs. Hence, if the capital project is underestimated, PCS costs will be similarly underestimated. A 2002 study involving 258 trans- portation projects collectively valued at $90 billion found that 86% experienced actual costs that were on average 28% higher than estimated (Flyvbjerg et al. 2002). That study concludes that “underestimation of costs at the time of decision to build is the rule rather than the exception for transportation infra- structure projects” (Flyvbjerg et al. 2002). If one applies the U.S. Army Corps of Engineers (U.S. ACE) supervision and administration rate of 5.6% (U.S. Army Corps of Engineers 1997) to Flyvbjerg’s sample, the PCS cost would be roughly $5.0 billion, a significant amount of money in any context. Using Flyvbjerg’s cost growth would mean that the agencies delivering these projects would be short $1.4 billion in the preconstruction phases of project development. The fact that project scope and quality are defined during the planning and design process leads one to infer that poor estimating accu- racy is actually robbing the project of proper resources to complete a thorough preconstruction process and perhaps ultimately results in imperfect construction documents that actually become the basis for construction cost growth after contract award (Molenaar 2005). A study by Carr and Beyor (2008) reported that consultant design fees have not kept pace with inflation for the past three decades. This creates a situation where “the high-quality pro- fessional services rightfully expected by the public will become increasingly difficult [to attain] if the erosion in fees continues unabated into the future” (Carr and Beyor 2008). In essence, this pricing pressure forces engineers to literally furnish the req- uisite level of design services with a steadily decreasing amount of resources. This could unintentionally induce a bias toward minimizing planning and design activities to maintain neces- sary project profitability, which in turn would manifest itself in the form of declining quality of construction documents. This environment is further exacerbated by the recent demand by owners to compress project delivery periods via programs like the FHWA’s Every Day Counts. A survey by the Construction Management Association of America found that the “demand for increasing speed of project delivery is the top reason for decline in construction document quality” (Construction Management Association of America 2003). The survey also reported that: In their responses to questions about the quality of construc- tion documents, more than half of the owners surveyed responded that these documents often have significant amounts of miss- ing information. Specifically, 45% of respondents indicated that construction documents, while sufficient, still had “significant information needed,” while an additional 12% found that docu- ments were typically inadequate because of major information gaps (Construction Management Association of America 2003). A number of studies have looked at the relationship between design quality and subsequent construction contract modifica- tions. Studies by Morgen (1986) and Kirby et al. (1998) found that design deficiencies are the major cause of construction contract modifications and that 56% of all modifications are aimed at correcting design deficiencies. Additionally, a study by Burati et al. (1992) found that deviations due to design errors discovered during construction account for 79% of all modification costs and average 9.5% of the total project cost. Thus, research has shown that improving planning and design quality has the potential to accrue benefits through reducing construction cost growth. A study completed for C H A P T E R 1

4the Oklahoma Turnpike Authority (OTA) confirmed this inference and demonstrated for one agency that, to a point, increases in actual construction costs compared to the early estimate were inversely proportional to the amount of money allocated for PCS (Gransberg et al. 2007). Figure 1.1 comes from that work and illustrates the relationship for OTA design–bid–build (DBB) bridge projects, specifically for the design fee expressed as a percentage of construction costs. The figure illustrates that, within the limitations of the research, providing adequate funding to properly complete PCS gives the agency more control over the final cost of the project. Said another way, an accurate PCS cost estimate increases cost certainty for DBB projects. This conclusion is confirmed by a recent study that found “DB [design–build] and CMGC [construction manager/general contractor] display lower cost growths than DBB and therefore provide greater cost certainty” (West and Gransberg 2012). Given this discussion, developing a pragmatic system with which to estimate PCS costs will promote final design qual- ity by reducing construction document errors and omissions (Carr and Beyor 2008; Construction Management Associa- tion of America 2003) and will accrue an immediate benefit by enhancing cost certainty for projects delivered using both traditional and alternative delivery methods (Gransberg et al. 2007; West and Gransberg 2012). 1.2 Problem Statement The first expenses borne in all projects are the costs to per- form planning, programming, and preliminary engineering. Construction uncertainty is at its absolute highest level, mak- ing the practice of setting a budget for PCS costs using a per- centage of construction costs merely the act of multiplying an arbitrary number by an estimated figure that will change as project development progresses. Hence, in many cases, the budget for developing a given project is effectively more uncertain than the budget for the project itself. To exacerbate the problem, research has shown that 86% of the time, the initial construction estimate and subsequent estimates are too low (Flyvbjerg et al. 2002), which means that the budget for PCS costs will also be too low. The phrase “you get what you pay for” applies in this situation. The amount of effort that can be applied to quantifying the cost of the project’s scope of work is limited by the available budget, and the inaccurate PCS cost estimate becomes a design quality issue, with in- house engineers and department of transportation (DOT) preliminary engineering consultants forced to make the time spent on refining the design fit the available budget. The final product is often a set of poorly prepared construction docu- ments detailing a product that is functionally overdesigned because the designers did not have the budget to produce a fully optimized design (Carr and Beyor 2008; Construction Management Association of America 2003). The state of the practice in PCS cost estimating ranges widely among DOTs. At times the variation is present within a single agency for different types of services and different stages of project development. Issues such as the range of design alternatives to be analyzed; the impact of environmental per- mitting, construction safety, and options for traffic control; and construction phasing to meet construction financing and budget constraints all make PCS cost estimating challenging at best and nearly arbitrary at worst. Therefore, the need for standardized guidance for estimating PCS costs is critical for DOTs to achieve the transparency, accountability, and fiscal responsibility that come with the tighter budgets experienced in the past several years. Hence, the objective of this research was to develop, test, validate, and package an accurate, consis- tent, and reliable method for estimating PCS costs. 1.3 Research Objectives and Tasks The NCHRP Project 15-51 request for proposal (RFP) states: The objective of this research will be to develop a guidance document on cost estimating for preconstruction services. The guide will address cost estimating for all types of preconstruc- tion services, whether performed by agency staff or consultants, addressing particularly issues specific to engineering and design services required for highway improvement projects (for exam- ple, surveying, preliminary engineering, environmental impact projection and mitigation planning, final design engineering). The guide will also address agency policies, procedures, and sup- port systems that will enhance an agency’s cost-estimating and management practices. To accomplish the stated objective, two sub-objectives were established to guide the research plan: 1. Identify, analyze, and understand the current models for PCS cost estimating; and Figure 1.1. OTA bridge projects’ cost growth from the initial estimate versus design fee (Gransberg et al. 2007).

5 2. Develop a guidebook for agency implementation of a stan- dardized approach to estimating PCS costs for construction projects. Accomplishing these objectives yields a PCS cost-estimating model that is specifically adapted for DOT projects and is not a repurposing of models in use in private industry. The spe- cific model is flexible enough to be tailored for implementa- tion within the statutory constraints of a given jurisdiction and is responsive to the concerns for equity and transparency of a state’s design and construction industry partners. The research has produced the following deliverables: 1. A guidebook for initiating and implementing a PCS cost- estimating system for highway projects at transportation agencies, 2. A research report that addresses the implications of adopt- ing the guidelines and barriers to implementation, and 3. An effective practices and tools report that documents find- ings that could be implemented before the final guidebook was produced. 1.4 Research Framework The research framework was derived from the NCHRP RFP, which was logically divided into three phases: • Phase 1–benchmark PCS cost-estimating practice, • Phase 2–develop and implement PCS cost-estimating method, and • Phase 3–furnish technical support to the AASHTO Sub- committee on Design (SCOD) during guidebook review and balloting. The outcome of the research is a guidebook and this research report based on a rigorous analysis of a state-of-the-practice review updates of past work on similar projects. The state of the practice then functions as a baseline from which the new contributions to this area are built. Phase 1 has comprehensively identified and categorized the PCS cost-estimating models that are currently in use. A significant barrier identified during this phase was the poor quality of collected PCS cost data and the lack of confi- dence held in it. This finding resulted in the need to alter the research plan for this project. Instead of prescribing a single PCS cost-estimating model, the research would need to show how to collect, clean, and properly maintain databases along with providing models for different applications depending on agency needs. Phase 2 investigated data-driven models and refined three stochastic techniques for PCS cost estimating along with a functional-level approach for resource management. The prod- uct of this research was used to create NCHRP Report 826: Estimating Highway Preconstruction Services Costs, Volume 1: Guidebook. The topical content was validated by the NCHRP panel and furnished to the NCHRP project oversight panel for review and approval. Phase 3 was to furnish technical support to the AASHTO SCOD during review and balloting of the guide. 1.5 Task Descriptions 1.5.1 Phase 1: Benchmark the State of the Practice As shown in Figure 1.2, during Phase 1 the research team evaluated current applications of PCS cost estimating in transportation and vertical construction industries. It also evaluated the state of the practice with respect to parametric cost-estimating theory and the way it is applied on a variety of project types. Due to the interdependent nature of the tasks in Phase I, the research aggressively overlapped in Tasks 1 and 2, and much of the work was performed concurrently in accordance with the work-effort assignments. The output of the literature review and the screening survey of AASHTO SCOD members was synthesized and documented in Task 2. Task 1. Define the state of the practice in PCS cost estimat- ing for transportation projects through a comprehensive litera- ture search and collection and analysis of relevant preliminary engineering and ICE consultant procurement documents, design contracts, relevant DOT policy/guidance documents, and a screening survey issued to AASHTO SCOD members at 2013 meeting. Select case study agencies and projects for Task 2. The literature review and content analysis from NCHRP Project 10-85 in the area of PCS costs for CMGC projects was updated and expanded to include the full suite of project delivery methods. Barriers to implementation from the litera- ture were identified, and information regarding PCS estimat- ing cost models and contingency development was collected. A methodology for developing a rational contingency for consultant design contracts was developed. Pre vious work on DB design administration costs was extended to cover DBB and CMGC. As a result, the final guidebook now covers the full spectrum of DOT project delivery requirements. The research team was able to move immediately to the development of a coding structure for data collection and characterization. The final coding structure permitted map- ping of both cost-estimating system and project delivery char- acteristics. The team modified the CMGC preconstruction services contract pricing model developed in NCHRP Proj- ect 10-85 as a basis for mapping. The second stage of Task 1 involved a screening survey con- cerning the use of PCS cost-estimating systems and variants for specialty items such as right-of-way (ROW). The team developed a short, comprehensive questionnaire.

6The survey for this study was directed at the members of the AASHTO SCOD to the various PCS cost-estimating practices and identify both potential case study opportuni- ties. The entire team contributed to the survey. Finally, a set of case study projects and case study agencies was assembled for use in Task 2. Task 2. Prepare case studies of PCS cost estimating at trans- portation agencies. After conducting the case studies, conduct pattern matching analysis between the case studies and deter- mine effective PCS cost-estimating practices, methods, and tools to be included in the guidebook content. The initial step in Task 2 involved consolidating and docu- menting the Task 1 information that would add value to the guidebook. Ultimately, the documentation was used as a basis from which questionnaires for the case study structured interviews were built. The first step involved assembling the case studies identified in Task 2 and filtering to ensure that the case study population covered the full spectrum of the research interest. The goal was to have a set of possible case study projects that furnished these attributes: • Range of project types–roads, bridges, tunnels, intelligent transportation systems (ITSs), vertical, and so forth; • Range of project sizes–typical small project to mega-project; • Range of project complexities–simple to highly complex; • Range of project locations–regionally dispersed; • Range of project delivery methods–DBB, DB, CMGC, and so forth; and • Other factors that may be found in Tasks 1 and 2. Figure 1.2. Phase 1 research plan.

7 Once the potential case study population was developed, the final list and the rationale for selecting each case were submitted to the NCHRP panel. On receipt of the panel’s agreement, the case study data collection began ahead of schedule. To achieve the objective of this task, nine indi- vidual case studies were conducted on projects using pre- dominantly DBB project delivery. Other delivery methods were also investigated on a smaller scale. The case study pro- tocol followed the guidance provided by Yin (2008). Case studies are empirical inquiries that investigate contempo- rary phenomena in their real-life context. The research team strongly believes that to adequately evaluate how the various agencies have successfully implemented PCS cost-estimating methods, case studies must be conducted. These are the pri- mary efforts needed to accomplish this objective: 1. Develop a case study protocol for conducting the case study interviews, 2. Conduct the case study interviews, and 3. Document the raw information collected and integrate it with data from the literature review. The key step in Task 2 is the first one: develop a case study protocol for the case study interviews and data-collection plan. In the proposed multi–case study analysis, the final protocol determined how the case studies were conducted, who the case study informants were, what information was collected, and how it was analyzed. The case study protocol followed rigorous qualitative research design and analysis methodologies based on Eisenhardt (1989, 1991), Yin (2008), Miles and Huberman (1994), and others. The protocol included a research synopsis of objectives, projects, field procedures that detail the logisti- cal aspects of the investigation (such as permission to access projects for data collection), interview questions, and docu- mentation to collect, as well as a format for documenting and analyzing the individual case studies (for internal research team distribution) (Eisenhardt 1989, 1991; Yin 2008). In addition, a plan was developed for cross-case comparisons to determine similarities and differences between cases (Eisenhardt 1989; Miles and Huberman 1994). Use of a case study protocol permitted the research team to conduct case studies separately in different parts of the country while maintaining the reliability of the case study results. Inter- nal validity was addressed by attending to multiple sources of evidence, and the use of multiple case studies improved the external validity of the project delivery and project control tools that were identified as promoting project success. The protocol included different categorizations of project char- acteristics, such as project procurement methods, payment provisions, and entity involvement in project development (managerial, engineering, and so forth). The protocol design also solicited data on barriers to implementation and methods and tools to overcome these barriers. Task 3. Prepare an interim report presenting the results of Tasks 1 and 2. The interim report will also include an updated work plan for the remaining tasks. The objective of Task 3 was to produce a comprehensive summary of findings and conclusions from Tasks 1 and 2, as well as an updated work plan for Phase 2 of the research. The team began planning, formatting, and categorizing prior to preparing the report itself. The team also applied that structure to keep the oversight panel informed through the monthly and quarterly reports. The next step was the development of a detailed interim report outline, based on the panel’s feedback from the quar- terly reports. The outline was used to guide the preparation of the report and assign responsibilities for drafting specific chapters or sections of the report. The report is comprehensive and describes in some detail methodology and results used to complete Tasks 1 and 2. It was submitted to the NCHRP panel on February 28, 2014. 1.5.2 Phase 2: Develop and Implement PCS Estimating Method Phase 2 entailed the research team creating a fully imple- mentable practice document that could be revised as required by local transportation agencies to align constraints and pref- erences as shown in Figure 1.3. It included the development of data-driven models for estimating PCS costs. Task 4. Prepare a guidebook for initiating and implementing a PCS cost-estimating system for highway projects at transpor- tation agencies. Since Task 3 resulted in an interim report, the major findings and highlights of the information were combined to produce a white paper containing a short synopsis of emerging findings. The process defined an effective practice, method, or tool as the intersection of two independent streams of information. In other words, the protocol for concluding that some practice is effective is that it was found in the literature, and its effec- tiveness was verified by either survey or case study evidence showing that it has actually been used successfully in the field. The white paper was entitled “Effective Practices and Tools for Estimating Preconstruction Service Costs” and was sub- mitted to the panel in October 2014. The paper also served as an in-progress review that could be disseminated with NCHRP permission to those agencies that need immediate guidance. The primary objective of Task 4 was to develop the guide- book for implementation of the data-driven PCS cost- estimating models found in this research. The task began

8Figure 1.3. Phase 2 and 3 research plan. ahead of schedule after the presentation of Phase 1 results to the AASHTO SCOD meeting in 2014. After assembling feedback from that event, the team continued with the develop ment of a PCS model. After collecting case study data, the research team began developing databases. As the team reduced, cleaned, and col- lated the data, it found it difficult to be able to guarantee that the data were complete and accurate. A number of inconsistencies were observed within collected data sets, and numerous blank cells indicated that some information was not complete. These issues of data quality and data availabil- ity created a number of challenges for the research team. The outputs of any model are only as good as its input, placing the quality of any estimating model produced into question.

9 After becoming familiar with the Utah and Iowa data sets, the team became concerned at how different the two sets were. Such variation between the agencies’ classification of data indicated that it would be difficult to achieve a consistent result between the two DOT’s models. These observations were reported to the panel in the Octo- ber 2014 quarterly report. The panel was notified that the research might ultimately lead to a process model that must be customized specifically to the way data are collected in a specific DOT rather than a fairly generalized approach that could be used by all DOTs. As a consequence of these findings, the direction of the research was modified slightly. Instead of creating a single PCS cost-estimating model, three different data-driven mod- els would be developed and complemented with a cost- estimating process. This allows DOTs to maintain their own data-collection and administration processes. 1.5.3 PCS Cost-Estimating Model In many of the case study projects, the DOT personnel expressed doubt regarding the accuracy of the available data associated with preconstruction activities. Some of the DOTs have a sophisticated process for collecting the PCS data, but in all cases, the data depended on the diligence of individ- ual employees to accurately reflect the distribution of the hours charged to a given project in a normal day. Addition- ally, despite these sophisticated data-collection mechanisms, DOTs lack the tools to process these data into meaningful information to support decision-making procedures during planning and design, including the development of reliable PCS cost estimates. In order to address this issue, this study presents a framework for the development of data-driven PCS cost-estimating models. This framework covers the entire development and application process, from an initial require- ments analysis of new models to the monitoring, control, and continuous improvement of existing estimating models. The PCS cost-estimating guidebook resulting from this study describes this framework in detail and explains how it can be used with three different PCS cost modeling methodologies: artificial neural networks, multiple regression analysis, and decision trees. Likewise, this study recognizes that the need for PCS cost estimates and the estimating capabilities of the available proj- ect data vary throughout the project development process. Thus, the guidebook defines and describes three types of esti- mates intended to fulfill needs at different levels: top-down, bottom-up, and functional-level estimates. Top-down esti- mates are conducted with very little information early during the project development process and are aimed to support strategic decision making. A bottom-up approach provides more precise estimates at the project level, but it requires more detailed project information, making it only available after investing some planning and design efforts. Finally, functional-level estimates (a type of bottom-up estimate) refer to the forecasting of costs or labor hours within each work area involved in the project (e.g., structural, environ- ment, geotechnical). Thus a bottom-up estimate may be per- formed by the aggregation of all functional-level estimates. 1.5.4 PCS Cost-Estimating Guidebook Development The guidebook explicitly describes the business case for making the change, discusses the barriers to making the change, provides a tool for structuring the PCS cost model to fit specific agency constraints, and provides tools for imple- mentation. With this wide variety of audiences and goals, the guidebook can only provide guidance. It is not a how-to textbook for all agencies to apply directly. The guidebook is written to give readers the necessary guidance for their individual roles in the development and adoption of PCS cost-estimating models in their agencies. An initial draft of the guidebook was submitted to the NCHRP panel on December 1, 2014. Reviews from the panel were related to the level of accuracy DOTs could expect from PCS cost-estimating models and the complexity added to the guidebook by including the description of some research instruments and computational details. As a result of this, the format of the guidebook was dras- tically modified. Special efforts were invested in the second draft of the guidebook to explain the major factors affect- ing the accuracy of data-driven PCS cost-estimating models. Several suggestions have been made to optimize the effec- tiveness of these estimates. These suggestions include tips to improve data management practices, the implementation of a PCS cost monitoring system, and the formalization of continuous improvement practices to progressively enhance the estimating capabilities of the models. Additionally, com- putational details and other complex technical content have been removed from the guidebook and placed in appendices, as suggested by the project panel. The updated guidebook also included an additional chapter on functional-level esti- mating since this bottoms-up approach that can be applied on the ground by departmental managers was not previously addressed, and it was deemed important to provide holistic guidance to agencies. Task 5. Conduct review and vetting in the field. Task 5’s objective was to test the applicability of the draft PCS cost-estimating guidebook. Vetting workshops of the guide- book were conducted with Iowa and Montana DOTs. A report of this process is detailed in Section 5.3. The feedback gained

10 from the workshops was used to further tweak the guidebook before its final review by the panel. Task 6. Prepare a revised guidebook (Interim Report 2) and a final report documenting the entire research effort. Due to the substantial changes made to the guidebook since the NCHRP panel’s initial review, the research team proposed that the guidebook be submitted separately from this final research report to give the panel time to provide feedback on the numerous modifications. The key deliverable of this task is the guidebook. It was therefore imperative that it be appropri- ately reviewed. A response to panel members’ review comments was submitted at the same time as the report, but were con- tained in a separate document. The final research report (this document) presents a summary of the entire research effort. Task 7. Furnish technical support to AASHTO Subcommittee on Design for review and balloting of the guidebook. The research team anticipated that the details of Task 7 would be developed as a part of the Task 6 panel review, and that the AASHTO SCOD would make known its support requirements before this task began. At this writing, the team expects that the majority of the technical support would be in the form of answers to SCOD member requests for infor- mation. The team believes that this could be served by pre- paring two webinars that would provide a forum to quickly disseminate the fundamental description and explanation of the PCS cost-estimating system. The first webinar’s subject would be a guided tour through the guide with a hypothetical example project designed to demonstrate the capability of the system. The second webinar would focus on implementation and would cover topics like training, resource requirements, documentation, data issues, and other similar topics. Finally, the team will make itself available during this period to pro- vide on-site presentations within the limits of the remaining project travel fund. 1.6 Report Format This final research report encapsulates all of the work com- pleted as part of this research project. Material from the interim report is included and built on to ensure a compre- hensive documentation of the entire project. • Chapter 1, this introduction, provides a brief background for this research project and functions as a guide to the rest of the report. • Chapter 2 is focused on establishing and documenting the current state of practice for preconstruction services (work from Tasks 1 and 2, the literature review, and initial screening survey). • Chapter 3 is a synopsis of the results of the PCS cost- estimating case studies and describes the types of data that were collected from each agency. • Chapter 4 is an explanation of the three data-driven PCS cost-estimating models and functional-level estimating method developed by the research team for implementa- tion at transportation agencies. This chapter summarizes Task 4. • Chapter 5 explains the vetting procedure required for Task 5 and its results. • Chapter 6 concerns the outcomes of the project and pro- vides recommendations for future research.