Estimating Highway Preconstruction Services Costs - Volume 1: Guidebook (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

53 6.1 Output Interpretation and Limitations The selected model is ready for application if it shows a satisfactory performance in accor- dance with the expectations of the agency. It is important to recognize the scope limitations of each model when applying it to real projects. For example, if a given DOT develops a model using historical preconstruction data from its previous corridor projects, the final selected model is only applicable to corridor projects awarded by this agency. Likewise, if an agency develops a model using data from previous paving projects completed in a given county, the model would be only applicable to paving projects to be executed in this county by this agency. Additionally, the output of a given PCS estimating model represents the cost of the same pre- construction activities covered by the observed PCS costs from previous projects used to build the model. Thus, if the historical data available to develop a PCS cost-estimating model only correspond to final design activities, outputs obtained from this model would be estimates of the cost of the final design preconstruction phase. It is important to understand that the data-driven models proposed in this guidebook are sensitive to changes in agencies’ and consultants’ preconstruction/design practices, including data-collection techniques. For instance, if a given DOT decides to move from traditional 2-D plans to 3-D modeling techniques, this change would be expected to have an impact on the PCS cost and the accuracy of a previously developed estimating model. Since some changes in preconstruction practices may not be easily perceived, such as variations in the quality or accu- racy of data-collection procedures as a result of staff changes, it is suggested to develop updated models on a regular basis with data sets that represent current preconstruction procedures, as described in the following section. 6.2 Continuous Improvement Since causal methods rely on historical data to estimate future values, PCS cost-estimating models can be constantly improved over time as more projects are executed and more data are collected, regardless of the model selected (multiple regression, decision tree, or artificial neural network). It is suggested to periodically repeat this framework procedure in order to optimize the model. The procedure may be repeated on a regular basis every 2 or 5 years (or a period of time that the agency considers convenient) or before this time if observed PCS cost perfor- mance measures do not meet the standard parameters established by the agency. Consequently, it is important to constantly track the performance of the model. All three methods should be considered when repeating this framework procedure since new data collected, changes in data-collection practices, or unperceived external factors may change the selection of the final model. For instance, the selection of an artificial neural network model today does not mean that this will always be the most suitable approach. C H A P T E R 6 Implementing PCS Cost-Estimating Models

54 Estimating Highway Preconstruction Services Costs 6.3 Use of Output as a Decision-Making Tool By definition, decision-making procedures involve the selection of the most suitable option from a set of alternatives based on the preferences and selection criteria of the decision makers. As a decision-making tool, PCS cost estimates can be used to select the design methods and technologies that best suit the needs and resource availability of an agency. For example, a given agency may decide whether to use in-house or external designers based on the expected PCS costs associated with each of these alternatives. Likewise, decisions can be made at the functional level (see bottom-up/functional-level approach in Figure 2.4) related to the design of specific project activities. For instance, the geotechnical engineer may use PCS cost estimates to deter- mine the cost implications of using 3-D technologies to model earthwork activities instead of traditional 2-D excavation and backfill plans. To make a comparison between these two design techniques, the model developed to estimate the cost of this work package must include a vari- able (probably a dummy variable) that indicates the design approach to be used. Regardless of the nature of the decision to be made, decision makers should take into consid- eration the limitations of the PCS cost-estimating models mentioned in Section 6.1. If the agency intends to compare the costs of two different design approaches, it must develop two different models following the procedures described in this guidebook. For example, in the case of com- paring the use of in-house versus external designers, the agency must develop two models: one using data from projects whose designs were completed with internal staff, and the other using data from projects whose designs were performed by external consultants. 6.4 Tracking PCS Cost Estimate Performance This section provides instructions on how to develop key performance indicators (KPIs) to measure the effectiveness of PCS cost-estimating models. This guidebook describes two types of KPIs used for two different purposes: measuring the performance of the model and track- ing the performance of a PCS cost estimate throughout the project preconstruction period. Table 6.1 describes the different KPIs proposed in this guidebook. Likewise, Appendix B: Project Monitoring – Preconstruction Services Progress, Part III, presents a template that may be used by DOTs to track and record values for these KPIs in a given project. 6.5 Capturing Lessons Learned The Project Management Institute (PMI) has a methodology for capturing lessons learned (King 2008). The King/PMI methodology consists of a series of questions that the project team should answer and record at the end of each project. These questions are related to three key areas: people, process, and product. Table 6.2 shows some examples of these questions by category. Answers to these questions may be directed to improve preconstruction practices or PCS cost- estimating models. Appendix B: Project Monitoring – Preconstruction Services Progress, Part IV, presents a template to assist project teams with the recording of their answers. 6.6 Implementing Database Maintenance and Model Development Within an Agency An agency may choose to maintain databases and develop data-mining models in any fashion that aligns most optimally with its resources and organizational structure. There are numerous approaches that can be taken. One possible system is to collect data and maintain databases from a central location. A centralized office may also be responsible for creating models with relevant

Implementing PCS Cost-Estimating Models 55 Table 6.1. PCS cost estimate – key performance indicators. At the Model Level The following KPIs are used to measure the overall performance of the PCS cost-estimating model. In order to draw any conclusions or take any corrective actions to improve preconstruction practices or the performance of the model, the agency should analyze the following three KPIs obtained from the application of the model in a series of projects. Corrective actions or model redevelopment may be needed only if one or more of these KPIs shows an average behavior that does not meet the agency’s expectations. Construction cost growth (CCG) (%) This KPI is intended to justify the use of PCS cost-estimating models as described in Section 1.3. It represents the variation, as a percentage, of the early construction cost estimate in comparison with the actual construction cost of the project. Final cost performance index (FCPI) This KPI measures the accuracy of the model by comparing the PCS cost estimate with actual PCS cost. Final cost of lost design effort (FCLDE) ($) The FCLDE corresponds to the total cost, in dollars, of activities associated with the development of discarded alternative designs. It also includes the cost of those portions of the original design that at the end were not used to construct the project. Lower FCLDE values represent a better utilization of the agency’s resources to perform final designs. At the Project Level The following KPIs are used to track the performance of the PCS cost estimate throughout the preconstruction period. These KPIs allow the project manager to detect anomalies in the performance of preconstruction activities and take corrective actions in a timely manner. Cost performance index (CPI) Unlike the FCPI, which is calculated at the end of the preconstruction period, this KPI compares the PCS cost estimate with actual PCS costs at any single moment during the preconstruction period. This indicator can only be determined in bottom-up estimates by comparing the estimated cost of completed work packages with the actual cost incurred by the agency to perform this work. Cost of lost design effort (CLDE) ($) Unlike the FCPI, which is calculated at the end of the preconstruction period, this KPI refers to the cost, in dollars, of activities associated with the development of discarded alternative designs at any single moment during the preconstruction period. A large value in this KPI may represent a poor definition of the project scope. Design placement (DP) ($) This KPI corresponds to the total PCS expenses incurred by the agency at any single point during the preconstruction period. This indicator is more suitable for top-down estimates since the lack of detail in these models does not allow the calculation of CPIs. The interpretation of this KPI is based on a comparison of its value with the total PCS cost estimate and the project manager’s professional judgment. Estimate at completion (EAC) ($) EAC is an adjusted estimate of the total PCS cost calculated from the known cost of completed work packages plus the expected cost of uncompleted work packages. This KPI can only be calculated for bottom-up estimates.

56 Estimating Highway Preconstruction Services Costs Table 6.2. Capturing lessons-learned methodology (King 2008). People Description Questions in this category should relate to team effectiveness and stakeholder interactions. Sample questions include those in the next cell. Questions • What did we learn about staffing—skills, knowledge, experience— that will help us on future projects? • What are the lessons learned about the issues that caused conflict among the team, and by the manner in which we resolved the problems and took corrective action? Process Description Questions in this category should relate to the inputs, tasks, and outputs of the project processes. Sample questions include those in the next cell. Questions • Were there any tools, techniques, or programs used on this project that should be used or avoided for future projects? • How effective was, or is, our data inventory? For whom, what, and when were these data collected? Product Description Questions in this category should relate to the project deliverables and success factors. Sample questions include those in the next cell. Questions • What is being done well or needs to be improved to define, evaluate, and ensure quality for the design? • What is being done well or needs to be improved to manage agency expectations? data for decentralized offices (counties or districts). This means a dedicated team with thorough knowledge of the models and data processes is responsible for all models, and a typical engineer need only input the key characteristics of a project into the model to obtain a cost estimate. Such an arrangement relieves the burden of training all PCS staff in data-mining techniques and ensures continuity of data capture and analysis across the agency.