Estimating Highway Preconstruction Services Costs - Volume 2: Research Report (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

19 C H A P T E R 3 3.1 Introduction There has been a substantial amount of research into esti- mating construction costs for highway projects, and there are a handful of studies about estimating design cost and prelimi- nary engineering, but there has been no research about esti- mating preconstruction services costs. The focus here is on the 16 projects and the nine agency case studies collected by the research team and the relevant analyses and observations of those case studies. Case studies formed the bulk of the origi- nal research conducted in Phase 1 of this research project and offered examples of PCS cost-estimating practices as well as agencies’ breakdown of the PCS information available within each agency. The chapter begins by discussing the case study data- collection protocol and methodology that allowed the team to secure information from each agency in a verifiable manner. This section includes a description of case study demograph- ics and the rationale for choosing each case study agency and the accompanying projects. Following the methodology section are condensed synopses of the case study summaries. Detailed case study summaries are contained in the appendices. Because of the large amount of information contained in the summaries, tabular summaries of relevant details are presented at the end of the summaries section to assist the reader in com- paring information from each study. 3.2 Case Study Protocol While the benchmarking survey conducted in Task 1 pro- vided some useful insights into the overall state of the prac- tice, the case studies were the primary source of data on the PCS cost-estimating techniques in Phase 1 and eventually became the basis for the practices suggested by the guidebook in Phase 2. This information on an agency’s PCS structure will also aid in the development of the parametric estimating model. Since the collection of information via agency inter- views and project case studies is the predominant research instrument in the research project, a large amount of time was invested to determine to how best to conduct the case studies, reduce the subsequent data, and capture valuable information. Researchers differ in their preference for research tech- niques and protocols best used in various environments; case study research has been shown to be a powerful research tool to evaluate and analyze emerging business practices such as PCS estimating techniques (Eisenhardt 1991). Case studies are particularly useful in answering questions about how things are done in detail, especially when examining a number of different cases (Yin 2008). The use of the case study method was essential in this research for capturing the unique nature and methods of the differing PCS cost-estimating procedures employed by each agency and understanding the rationale behind the agencies’ chosen methods. The major objection to the use of case studies has been the perceived lack of statistical rigor. Recognizing this criticism, the researchers sought to generate a defensible, repeatable method to guide the case study process. This method was formalized and recorded in the case study protocol for the project. Creation of the case study protocol was guided by an influential book on the technique written by Yin (2008). The case study protocol served to establish the purpose of the case studies and the research questions to be answered by them. Clearly stating the specific information sought by the researchers at the start of this crucial task ensured that all researchers who were conducting case study interviews under- stood the ultimate goals of the research. The background infor- mation for the protocol included key sections of the project proposal and work plan, such as the three questions used to further explore the objectives of this research: 1. What project characteristics are important to developing an accurate PCS cost estimate? 2. What steps must be followed to implement a standardized PCS cost-estimating methodology? Preconstruction Services Case Studies

20 3. How are PCS cost-estimating consultant contracts suc- cessfully procured? The most important aspect of the protocol was the field data-collection procedures. These procedures standardized the method to conduct all of the case study data and facilitate consistent and comparable results among the case studies. The key research instrument is the structured interview based on a standard case study questionnaire (U.S. Gov ern- ment Accountability Office 1991). The questionnaire was sent to the participants a week in advance of the interview. Each agency’s PCS estimating procedures are unique, and the interview process was designed to capture that uniqueness while generating a standard comparable output. To that end, the questionnaire maximized the use of yes/no questions and matrices of checklists to be complete for every case study. Additionally, open-ended questions were crafted to generate in-depth discussion to fill in the details that surveys and ques- tionnaires cannot easily capture. 3.3 Case Study Process The case study protocol included a pilot case study to eval- uate the efficacy of the process before modifying the case study protocol and completing the remaining cases. The pilot study also served to allow the research team an opportunity to become familiar with the case study protocol for this proj- ect and provide comments on it or recommendations for changes. After the pilot study took place with the Montana DOT, a few minor adjustments were made to the agency struc- tured interview questionnaire. There were additional expla- nation boxes added to gain a more in-depth understanding of each agency’s PCS processes, and both loss of design effort and geographic factors were added to the matrix concern- ing the list of factors that influence the PCS estimate. Finally, there was the addition of a question about the impact the DOT thought a better PCS estimate would make on the plan- ning process. The case study protocol for this project mandated a specific sequential order for communications and interactions with project participants that was followed for each case study. First, all interviews with the participating agencies were conducted on site and in person at the agency’s headquarters to ensure appropriate people were available to answer the questions provided. Other initial inquiries were made via email, but the personal contact was vital to the quality of the informa- tion collected in each case study. The personal contact with the key PCS cost-estimating personnel participants provided a champion for the research effort and a specific point of con- tact for queries during data reduction and interpretation. The participants were not compensated for their time by the research team, making it essential to secure at least one agency staff member who was enthusiastic about assisting with the research effort and was in a position to coordinate with the rest of the agency. 3.4 Case Study Selection As this was a national research project, the research team wanted a fair representation of states considering factors like population, budget, land area, and in-house versus outsourced PCS makeup. There was an original shortlist of 16 states pro- posed to the panel in the kick-off meeting on April 17, 2013. After a discussion with the panel, nine DOTs were selected. Four were selected as agency case study states where all the cost data for multiple projects were captured for use in Task 4. These four agencies along with the other five were all project case study agencies where data were collected on the agencies’ PCS cost-estimating procedures, and some project case study projects were collected. The nine participating agencies are listed in Table 3.1 along with population, land area, and the DOT’s yearly construction budget. Agency Area Population (million) Land Area (square miles) Budget ($ million) California* 38.3 155,779 $13,000–$15,000 Colorado 5.27 103,642 $500– $700 Iowa* 3.09 55,857 $400 Maryland 5.93 9,707 $600–$800 Montana 1.02 145,546 $385 New York* 19.7 47,126 $1,000 Oklahoma 3.85 68,595 $632–$790 Rhode Island 1.05 1,034 $300 Utah* 2.90 82,170 $1,100 *Indicates agency case study state where data for all projects from the previous 5 years was collected. Table 3.1. Case study agency information (U.S. Census Bureau 2014).

21 Representatives from these nine agencies were interviewed in a structured interview process to determine the agencies overall PCS cost-estimating procedures. The interview tem- plate is shown in Appendix C. Each agency was also asked to provide two to five projects for case studies for the research. The researchers ended with 16 projects from six of the nine agencies. Figure 3.1 shows the geographical distribution of the states. A synopsis of these interviews and case studies is provided in Section 3.5. 3.5 Case Study Agency Synopsis For full case study reports refer to Appendix D. 3.5.1 Agency Case Studies Agency: California Department of Transportation–Caltrans Data Collection Details • Collected Caltrans collects project cost data for PCS through engi- neers’ timesheets. Caltrans uses data collected from past proj- ects to estimate the PCS cost for future projects. It also has a system called PIPE scan, which is used as a starting point for PCS estimates. Current methods used to estimate PCS costs for a project include a direct estimate of hours as well as an average percent support-to-cap ratio. Caltrans performs 90% of PCS in-house and contracts out 10% of PCS. Each district has its own on-call contracts with preselected qualified architect and engineering con- sultants. Caltrans can outsource all PCS activities except advertisement for bids, evaluation of bids, and award of contract. It is rare for Caltrans to outsource PCS concern- ing cost estimates, ROW plans, and ROW utility acquisition and relocation. At Caltrans, if there is a loss of funding for a project when it is in the PCS phase, the project will be terminated. Once funding for the project is resumed, a new project number is assigned; therefore, it does not consider loss of design effort. These costs will be included in the overhead rate. To improve PCS estimates, Caltrans believes it needs a better model for historical data analysis and needs to do bottom-up estimates. At this stage, the project manager does not control the people working on projects in the PCS phase. Caltrans believes that having more accurate PCS cost estimates would have some impact, mainly on the budget process. Agency: Iowa Department of Transportation–Iowa DOT Data Collection Details • Collected • Approximately 1,303 projects • 11 project types • Figure 3.2 shows an example of the data collected for Iowa DOT projects. Figure 3.1. Geographical distribution of the case study states.

22 Iowa DOT collects project cost data for PCS through engi- neers’ timesheets. These data are collected and stored mainly for accounting purposes. Iowa DOT does not use data col- lected from past projects to estimate the PCS cost for future projects. It does not estimate PCS cost for a project. Iowa DOT can use both in-house and on-call consultants; it also uses other consultants, but only for larger, less-common projects. Currently, Iowa DOT is not estimating PCS cost for proj- ects, but it is looking to adopt this in the future. To improve these estimates, Iowa DOT believes it needs to learn how to use the data it already has. Iowa DOT has been capturing PCS hours for a few years, and it needs a way to organize these data to make them useful in PCS estimating. Iowa DOT thinks that having a more accurate estimate of PCS costs would have a large impact on the planning process for the agency and would allow it to better budget staff time. It also would be valuable to know the number of hours per task and to be able to compare these to consultant design hours. Agency: New York State Department of Transportation–NYSDOT Data Collection Details • Collected Figure 3.3 and Table 3.2 show examples of the data avail- able for NYSDOT projects. NYSDOT collects project cost data for PCS through engineers’ timesheets. These data are used by Figure 3.2. Snapshot of Iowa DOT project data.

23 Figure 3.3. Snapshot of NYSDOT project data. Project name Western Ave – NYS I787 NYS Procurement method DBB DBB Project type Reconstruction Bridge rehabilitation PCS performance Consultant In-house Total project cost $9,700,000.00 $28,000,000.00 Total PCS cost $1,280,000.00 $1,333,346.08 PCS percentage 13% 5% Complexity 2 4 Sub-consultants 2 0 Lanes 6 6 Plan sheets 198 648 NEPA classification Cad X Cad X Bridges 0 6 Highway classification Interstate Interstate Length of project (miles) 12.1 4.3 Table 3.2. NYSDOT project data.

24 project managers to predict an estimate for future projects with similar qualities. NYSDOT uses an in-house system called DPR that contains a selection of tools to estimate PCS hours. NYSDOT is looking at moving to the use of Primavera P6 in the future. By dollar value, NYSDOT performs 50% of PCS in-house and 50% is outsourced, and by number of projects, 90% is in-house and 10% is outsourced. NYSDOT does not perform environmental sampling and testing or surveying services; it uses on-call contracts for these services even if all PCS ser- vices are performed in-house. NYSDOT can outsource all PCS activities except advertisement for bids, evaluation of bids, and award of contract. NYSDOT does not consider number of plan sheets as an influential characteristic in the PCS estimating due to recent advances in technology and the general move to electronic plans. NYSDOT believes that a major setback to estimat- ing PCS costs is how to estimate inflation as it is difficult if project development occurs over multiple years. NYSDOT believes that to improve its PCS cost estimating, it needs to move to task-based estimating, but it is skeptical about whether the time and effort would result in any real value for the agency. Agency: Utah Department of Transportation–UDOT Data Collection Details • Collected • Approximately 564 projects • 21 project types • Five procurement methods ≈ 516 design–bid–build • Figure 3.4 and Table 3.3 show examples of the data col- lected for UDOT projects. UDOT collects project cost data for PCS through engi- neers’ timesheets. These data are stored in ePM (electronic project management) and are used by project managers to predict an estimate for future projects with similar qualities. UDOT also performs a direct estimate of hours for PCS work, which is compared with the past project cost range as a check. By dollar value, UDOT performs 25% of PCS in-house and 75% is outsourced. UDOT can outsource all PCS activities except advertisement for bids, evaluation of bids, and award of contract. UDOT tries to decide early on whether the proj- ect will be outsourced or performed in-house so that it can set the budget early. Figure 3.4. Sample of UDOT agency project data.

25 UDOT does not believe it sets out to make mistakes; there- fore, it does not consider loss of design effort necessary in estimating PCS costs. To improve PCS estimating, UDOT believes it needs to retain, hire, or train new experienced staff. UDOT believes that having more accurate PCS cost estimates could have some impact on the planning process and allow them to refine allocation of resources and negotiate with con- sultants better. 3.5.2 Project Case Studies Agency: Colorado Department of Transportation–CDOT CDOT does not collect past project cost data for PCS. For federally funded projects, CDOT has to submit an indepen- dent project cost estimate, and in this case, 10% is used for PCS costs. CDOT will collect all project data for projects in the bridge enterprise program and also for large projects. Table 3.4 shows an example of CDOT project data. By number of projects, CDOT performs 45% of PCS in- house and 55% is outsourced. CDOT can outsource all PCS activities except advertisement for bids, evaluation of bids, and award of contract. CDOT does not have a policy on the amount of work outsourced; however, it needs to have rea- sonable justification before outsourcing a project. CDOT considers the construction cost of a project to be a major influence on the PCS estimate for in-house projects but only a minor influence if PCS will be contracted out. CDOT is looking to adopt a tool that can help it estimate PCS costs, especially as there is a loss of experience when it employs young engineers. To improve its PCS estimating, the agency believes it requires good tools as well as data that align with the systems already in place at the agency. An improved PCS estimate is likely to benefit budget portfolio manage- ment as people usually involved with these estimates are often not engineers but are planners. Agency: Maryland State Highway Administration–MSHA MSHA does record in-house PCS hours on a per-project basis. It records these hours using time-tracking software. MSHA uses data collected from past projects along with Project name Region 3 - UT Region 2 - UT Region 4 - UT Procurement method DBB DB DBB Project type Reconstruction Continuous flow intersections Rehabilitation PCS performance ROW – Consultants All other PCS in-house Consultant In-house Total project cost $4,200,000.00 $48,981,854.37 $2,260,000.00 Total PCS cost $277,253.92 $3,704,380.09 $17,634.00 PCS percentage 7% 8% 1% Complexity 2 5 1 Sub-consultants 1 4 0 Lanes 6 6 2 Plan sheets 98 115 0 NEPA classification Cad X SES Cad X Bridges 0 0 Highway classification Rural principal arterial Major arterial Major arterial Length of project (miles) 2.5 2 8.48 Table 3.3. UDOT project data. Project name Eagle interchange Procurement method CMGC Project type Major structure PCS performance Consultant Total project cost $15,100,000.00 Total PCS cost $1,510,000.00 PCS percentage 10% Complexity 4 Sub-consultants 8 Lanes 4 Plan sheets 515 NEPA classification Cad X Bridges 2 Highway classification Major collector Length of project (miles) 0.35 Table 3.4. CDOT project data.

26 standard percentages to estimate the PCS cost for future projects. The old system used 15% of the construction cost as preliminary engineering; MSHA now uses a curve system on preliminary engineering. Table 3.5 shows an example of MSHA project data. MSHA has a standing contract for a general engineering consultant (GEC). MSHA can perform the entire precon- struction process in-house and can also outsource all PCS except ROW utility acquisition and relocation, advertisement for bids, evaluation of bids, and award of contract. MSHA is currently estimating PCS costs for all projects. To improve these estimates, the agency believes it needs to develop a historical database of previous estimates. MSHA believes that having more accurate PCS cost estimates would have a large impact on the planning process since it believes that it would provide more efficiency to managing funds. Agency: Montana Department of Transportation–MDT MDT does record in-house PCS hours on a per-project basis. It records these hours using the engineers’ timesheets and has a time allocation system per job. MDT does not use data collected from past projects to estimate the PCS cost for future projects. Table 3.6 shows an example of MDT project data. MDT has a system that records past hours and durations of activities of 3 to 5 years to reconcile with activities to average activity hours. This system has no feedback loop, and there- fore it is not used to look at past projects or to re-access the activity hours in OPX2 (project management tool). MDT can perform the entire preconstruction process in- house except for feasibility studies, and it can outsource all PCS except advertisement for bids, evaluation of bids, and award of contract, which is considered in the construction department. Approximately 20% of the PCS program for MDT is outsourced. Currently MDT is estimating PCS costs for all projects using a standard percentage of construction costs. To improve PCS estimates, MDT believes it needs to get to function-based estimating, and it also needs to determine how to allocate the funds in split-corridor projects. MDT also believes that it needs to improve how it captures the hours on timesheets. Agency: Oklahoma Department of Transportation–ODOT ODOT does not record in-house PCS hours on a per- project basis. Approximately 50% of engineers’ time spent on PCS is billed to departmental overhead. Table 3.7 shows an example of ODOT project data. ODOT can perform the entire preconstruction process in-house except right-of-way acquisition. It can also out- source all PCS except preferred alternative, NEPA and permit approval, final plan package [RFP and request for quotation (RFQ)], advertisement for bids, evaluation of bids, and award of contract. Currently, ODOT believes estimating PCS cost would add value to the agency, but it has yet to implement a process to do so. To improve PCS cost estimates, the agency believes it needs to make direct changes to its projects and agency cul- ture. ODOT believes that it would be difficult to convince all people within the agency to adopt a PCS estimating system. Agency: Rhode Island Department of Transportation–RIDOT RIDOT does record in-house PCS hours on a per-project basis. It records these hours using the engineers’ timesheets. Project name Taneytown streetscape MD 924 Procurement method DBB DB Project type Reconstruction Safety PCS performance Total project cost $22,000,000.00 $10,000,000.00 Total PCS cost $2,200,000.00 $800,000.00 PCS percentage 10% 8% Complexity 4 4 Sub-consultants 8 2 Lanes 2 4 Plan sheets 354 NEPA classification Cad X Bridges 0 0 Highway classification Urban other principal arterial Urban arterial Length of project (miles) 2 0.5 Table 3.5. MSHA project data.

27 RIDOT does not use data collected from past projects to esti- mate the PCS cost for future projects. Design costs are esti- mated by using 15% of total construction cost. However, this is not uniform; smaller projects tend to be of a higher per- centage, and larger projects tend to be of a lower percentage. This process is only an educated guess. RIDOT does contract out PCS. It has several on-call con- sultants because almost all its design work is outsourced. It uses two consultants for highway work, two for bridges, and four for traffic engineering. No single firm is the dominant GEC. RIDOT can advertise for bids, evaluate bids, award con- tracts, and perform some ROW utilities acquisition and relo- cation; all other PCS processes are outsourced. RIDOT did not provide data for project case studies. RIDOT does not see value in estimating PCS costs. Since it is a small organization, it has yet to develop a database to keep track of and evaluate PCS costs. Its priority lies in esti- mating construction costs. To improve these estimates, the agency believes it needs a database to pull scattered records and documentation of PCS into one place. There is a 2-year Project name Alberton – MT Yellowstone – MT Richey – MT Libby – MT Manchester – MT Procurement method DBB DBB DBB DBB DBB Project type Rehabilitation Bridge replacement Reconstruction Rehabilitation Rehabilitation PCS performance In-house 61% in-house, 39% consultant 81% in-house, 19% consultant 69% in-house, 31% consultant In-house Total project cost $15,160,216.69 $11,117,526.18 $11,671,335.94 $5,154,041.00 $13,654,704.61 Total PCS cost $ 326,984.74 $ 1,350,022.32 $ 747,932.55 $ 523,441.08 $ 221,626.30 PCS percentage 2% 12% 6% 10% 2% Complexity 1 4 3 4 2 Sub-consultants 0 3 1 3 0 Lanes 4 2 2 2 4 Plan sheets 41 113 351 284 258 NEPA classification Cad X EA/FONSI Cad X Cad X Cad X Bridges 6 1 1 0 3 Highway classification Principal arterial Urban arterial/ rural minor arterial Major collector rural Major collector rural Principal arterial (freeway) Length of project (miles) 9.8 0.7 10.7 5.1 5.4 Table 3.6. MDT project data. Project name Garvin – OK Beckham – OK Payne – OK Procurement method DBB DBB DBB Project type Resurfacing Resurfacing/ Bridge rehab Pavement overlay PCS performance In-house Outsourced Outsourced Complexity 4 4 3 Sub-consultants 0 2 2 Lanes 4 4 4 Plan sheets 131 60 50 NEPA classification Cad X Cad X Cad X Bridges 6 5 0 Highway classification Interstate Interstate I-35 Length of project (miles) 6.5 7.93 5.4 Table 3.7. ODOT project data.

28 election cycle, so government and legislative representatives change regularly; therefore, projects continue to lose and gain importance depending on the political influence. RIDOT believes that having more accurate PCS cost estimates would have no impact on the planning process. It believes that PCS costs have very little impact on the overall program, and proj- ects will be executed no matter what the magnitude of PCS costs are. 3.6 Case Study Analysis The purpose of Phase 1 of the research project was to bench- mark the state of the practice and identify, analyze, and under- stand current models for PCS cost estimating. In Section 3.2, three questions were given to further explore the objectives of this research: 1. What project characteristics are important to developing an accurate PCS cost estimate? 2. What steps must be followed to implement a standardized PCS cost-estimating methodology? 3. How are PCS cost-estimating consultant contracts suc- cessfully procured? The case study analysis looks to answer these questions and furnish information on emerging trends within the DOTs. 3.6.1 What Project Characteristics Are Important to Developing an Accurate PCS Cost Estimate? A report from Williams et al. (2013) identified 12 project characteristics that are inherent in each project for NYSDOT and can be used to estimate design effort. These characteris- tics were evaluated by a team of NYSDOT and FHWA person- nel so are applicable to the target audience of this research. These characteristics were used as a base for identifying the project characteristics important in developing an accurate PCS cost estimate. Each agency was asked to fill in a matrix identifying which project characteristics had the most influence on the PCS cost estimate. The average rankings for these characteristics were analyzed using a t-test to determine the equality of the means of the responses and to categorize the factors into the three levels of influence. The results for the most influential factors from an agency’s perspective and then from a project perspective are shown in Table 3.8 and Table 3.9 respectively. Question V.1 of the interview was as follows: How influential do you think the following characteristics are in estimating the overall PCS cost for a typical design–bid–build project? 1–No influence 2–Some influence 3–Major influence The list of characteristics or influence factors provided was as follows: 1. Complexity, 2. Project type, Influence Factor Mean Response Tier 1 [2.56–3.00] 1. Complexity 3.0 2. Project type 2.89 3. NEPA classification 2.67 Tier 2 [2.00–2.56] 5. Length of project 2.56 6. Number of bridges 2.44 7. Number of plan sheets 2.33 8. Number of lanes 2.0 9. Geographical 2.0 Tier 3 [1.44–1.56] 10. Highway classification 1.56 11. Number of sub-consultants 1.44 12. Loss of design effort 1.44 Table 3.8. Influence factors ranked based on mean response values from nine DOTs. Influence Factor Mean Response Tier 1 [2.42–2.75] 1. Complexity 2.75 2. Project type 2.56 3. Construction cost 2.42 Tier 2 [1.92–2.07] 4. Number of bridges 2.07 5. Length of project 2.06 6. Highway classification 1.94 7. Number of sub-consultants 1.93 8. Number of plan sheets 1.92 Tier 3 [1.62–1.81] 9. NEPA classification 1.81 10. Number of lanes 1.63 Table 3.9. Project influence factors ranked based on mean response values from 16 projects.

29 3. Number of sub-consultants, 4. Construction cost, 5. Number of lanes, 6. Number of plan sheets, 7. NEPA classification, 8. Number of bridges, 9. Highway classification, 10. Length of project, 11. Geographical, and 12. Loss of design effort. The question was answered by the DOT representative dur- ing the interview, and later the same question was answered in the context of a specific project. The researchers collected nine sets of responses from the DOTs. The mean value of the response to each influence factor is given in Table 3.8. In these tables, the factors are ranked based on the mean response value. It is worth noting that no additional factor was suggested to be added to the list that was presented to the interviewees. The responses summarized in Table 3.8 represent the impor- tance of influence factors from the point of view of the state departments of transportation. The researchers have catego- rized the factors into three tiers. Tier 1 consists of the factors that DOTs felt had the most influence on PCS costs. Tier 3 consists of factors that scored an average of well below 2.0 and, hence, were considered to have little to no influence on PCS costs. A statistical analysis was conducted to see the effect of variability of responses to each factor and to see if there were significant differences between factors. As an example, if a factor has a mean response of 2.89, is this really different from another factor with a mean response of 2.67? In order to investigate this question, the researchers con- ducted a two-tailed t-test for comparison of mean responses. The null hypothesis was that the means for any of the two selected factors were equal. The alternative hypothesis was that the means were not equal. In general, for the factors in each tier of Table 3.8, one could not reject the hypothesis that the means were equal. This means that the factors within each tier will have more or less the same importance. There are some concerns with using this test for this application. First, the number of data points is only 9. Second, the assumption of normality is not realistic; however, the test provides an insight into the effect of variance on the possible values of each factor. So the main purpose for conducting these tests here was to have a systematic and consistent method to group these factors into the three tiers so that the most influential factors can be concentrated on. Table 3.9 gives the project influence factors ranked based on their mean score from 16 projects. The main difference between these factors and the factors listed in Table 3.8 is the respondent was weighing the influence of each factor against a specific project rather than the whole agency. This table does not include two of the factors listed in Table 3.8 (“geo- graphical” and “loss of design effort”). As can be seen, the most influential factors remain the same in both cases, with the exception of “NEPA classifica- tion,” which has been relegated to Tier 3 in Table 3.9. This may merit further consideration. “Number of lanes” scored higher at the agency level, while “highway classification” and “number of sub-consultants” scored significantly higher at the project level. Overall, while these two tables agree on many of the most influential factors, in Tier 3 factors there are some differences. The same statistical approach explained earlier for group- ing factors (two-tailed t-test for comparison of means) was applied to the factors in Table 3.9. The three tiers presented are the outcome of that analysis. In other words, the equality of factor means within each tier could not be rejected statisti- cally. This analysis was based on a sample that varied between 14 and 16 projects because not all respondents scored every influence factor. As a comparison between the outcome of Table 3.8 and 3.9, a correlation coefficient was calculated between the ranks of factors in each table. The rank correlation between these fac- tors was calculated as 0.60. A correlation coefficient of 0.64 was also calculated between the scores of factors in the two tables. In both cases, these values show that there is moderate correlation between the results of the two tables. There seems to be little doubt that the most significant fac- tors at the agency and the project level are the following: • Project complexity, • Project type, and • Construction cost. This was valuable information for developing the para- metric estimating model as these three factors could be used as the most influential input variables to estimate the PCS costs for a project. Project complexity is a subjective variable, making it difficult to incorporate this as in input variable; in Phase 2 the researchers looked to develop a complexity index as a way of standardizing this variable. One of the controversial characteristics that came from this analysis was the number of plan sheets. While some DOTs still do a lot of their work on paper, DOTs such as NYSDOT are moving more toward technological-based plans, making the measure redundant. This is also highlighted in a report by Tippett and LaHoud (1999), and Sturts and Griffis stated that “technology is revolutionizing the way engineers work and there is a need to revise the pricing strategies for engineering design services” (2005). Loss of design effort was a characteristic added by the researchers after the pilot study. This was defined as “design

30 Figure 3.5. UDOT’s project development process. work completed but not used in the final project due to changes in scope during the design process.” During the PCS phase, this is likely to occur often, especially if the project scope is not well defined. It occurs when there is a change in the scope that renders hours already billed to the project redundant; the work is still a PCS cost to the project and, therefore, should be accounted for in the estimate. When they were questioned about this influencing factor, it was clear that this concept was either not fully understood or not consid- ered by state DOTs. It was suggested by several interviewees that incorporating lost design effort into the PCS estimate was inappropriate because it indicates that the agency plans to waste valuable design time. The intent of this factor is to account for typical scope changes/refinements and human error that require reworking of the design. In the final analy- sis, the issue is moot since none of the case study agencies had a means of tracking lost design effort. 3.6.2 What Steps Must Be Followed to Implement a Standardized PCS Cost-Estimating Methodology? One major trend that needs to be addressed in order to implement a standardized PCS cost-estimating methodology is to standardize the terminology used by state DOTs. Within the nine states visited for this project, there was a confusing mix of terminologies used for different phases of the projects. It was useful to have the PCS project development process to give a standardized template for each agency to adapt to their own project development process. Figure 3.5 shows UDOT’s project development process and the estimating process for UDOT. This figure shows three deci- sion sections that act as roadblocks for the project. The esti- mate must be prepared and reviewed at these points before the project status can move forward. A similar process was seen in the literature for some agencies such as CDOT; how- ever, when the researchers visited the agency, they were informed that the published process was not in use. Having a project development process with milestones to ensure that engineers perform PCS estimates and then sec- ondary milestones to recheck these estimates is an effective way of continually improving estimates. 3.6.3 Types of PCS estimates Three basic types of estimates were found during the case study interviews: • Direct estimate of hours, • Standard percentage of construction cost, and • Past project cost range. Table 3.10 shows which method(s) each agency uses. The level of sophistication used within each department varies, but it can be seen that Utah is the only state currently using two methods and comparing the results. Two states were found to not estimate PCS costs at all, which is a trend found in the screening survey. Two types of estimates have emerged from the case studies: • Top-down (macro) estimates produced by an experienced estimator, useful for managers who have limited knowl- edge of the process to complete the project (Larson and Gray 2011); and

31 • Bottom-up (micro) estimates, which usually correlate to a work breakdown structure (WBS); each activity is esti- mated by the person who is involved with monitoring the project (Larson and Gray 2011). Table 3.11 has been adapted from Larson and Gray (2011), and it shows the project characteristics associated with choos- ing to do top-down or bottom-up estimates. A top-down esti- mate is defined as the use of a parametric estimating factor like percentage of estimated construction costs to determine the PCS budget. A bottom-up estimate calculates the number of labor hours estimated for each PCS task, calculates the average labor-hour rate, and rolls the cost up from these detailed esti- mates of effort. When the researchers met with Caltrans, two types of proj- ects were used for corridor-level projects (termed “parent projects” by Caltrans; see Section 2.2.1) when the costs for planning and scoping are incurred. The parent projects spawn a series of child projects or single projects (Section 2.2.1). This project structure was also found in the Iowa DOT case study data. The following question was put to the panel in the December quarterly report: Corridor projects versus single projects. In many cases, the team found that much of the PCS work was being done on a corridor basis rather than an individual project basis because the DOT did not know the level of available funding. Once the funding was identified, the work was then split into phases/ separate projects, and final PCS costs were then expended only for those phases/projects where construction funding was going to become available. Thus, attempting to separate the cost to com plete planning, environmental, survey, right-of-way, etc. is impossible. We need guidance on how the panel thinks we should deal with this issue. The simple solution, which is our recommendation, is to only conduct further work on projects that stand alone and for which we have identifiable PCS cost data. This limits the results of the research, specifically, the Task 4 model to be applicable to only those types of projects. The bigger question relates to who performs the estimate and the data used within the agency’s project development process. The initial estimate of PCS costs for the parent proj- ect is typically done at the programmatic level. Therefore, Table 3.11 shows that a top-down estimate is more appropriate as it looks at the bigger picture. This estimate is useful for allo- cating program funds, making it valuable to regional or pro- gram managers. When the project moves into the child-project phase, the scope is likely to be better defined and require a more detailed estimate. In this situation, a bottom-up esti- mate is likely to be more appropriate (Table 3.11). The greater detail used in this estimate will more closely portray the level of effort required to complete each major task and will provide a more accurate PCS cost estimate around which to establish the budget for the PCS teams and the project managers. Once a child project is generated from the larger parent project, Caltrans allocates a percentage of the costs incurred during the parent project stage to the child project. A varia- tion of this was seen in an NYSDOT project. The project was initially procured as a large project; however, due to fund- ing constraints, the project was changed to a much smaller project. In this case, the PCS for the project had already been awarded to a consultant, and this project ended up with a much higher PCS cost percentage than other projects within the agency. This also links with feedback the researchers received from NYSDOT that a way to improve its PCS estimate would Method Agency Direct estimate of hours • California • Utah Standard percentage of construction costs • Colorado • Montana • Rhode Island Past project cost range • Maryland • New York • Utah Do not estimate PCS costs • Iowa • Oklahoma Table 3.10. Agencies’ PCS estimating methods. Condition Top-Down Bottom-Up Strategic decision making X Cost and time important X High uncertainty X Internal small project X Fixed price contract X Details needed X Unstable scope X Table 3.11. Conditions for preferring top-down or bottom-up estimates (Larson and Gray 2011).

32 be moving into top-down estimates to make more informed decisions at the programmatic level. 3.6.4 How Are PCS Cost-Estimating Consultant Contracts Successfully Procured? This section is derived from the literature review and shows all the states that have developed models to determine the cost of outsourcing design. A report by Ismail and Sutliff (2011) for Caltrans showed that out of the three states that have invested and developed these models, none could con- firm that these processes were being implemented. Table 3.12 has been adapted from this report and shows the model and its use. This was also apparent in the case studies as all nine states responded that they do not compare the cost of per- forming work in-house to the cost of outsourcing as part of the outsourcing decision process. A report for the Louisiana Department of Transportation and Development found that outsourcing design cost was about 20% more expensive than performing the work in-house (Ismail and Sutliff 2011); however, this was not the same for all agencies. When asked why the agency chose to outsource PCS costs, all DOTs responded that staff availability and special expertise were the two main reasons. Other reasons that were mentioned were expedited project delivery, to strengthen the local economy, transfer design liability, and ability to release DOT personnel to perform PCS contract administration duties. It was com- mon practice among DOTs to outsource larger, more com- plex projects. It can be argued that this research will benefit DOTs more for outsourcing projects as DOTs can use the estimate as base for negotiations with the designer. However Iowa DOT indicated that it would adopt a model to aid with in-house resource allocation and help better identify shortfalls in resources. A possible option is to use the model to determine when it is necessary to employ consultants on work to avoid overloading DOT staff. This is becoming an increasing issue within DOTs as there is a move in many states to downsize the government staff. For example, UDOT’s staff decreased from 3,500 employees in 2000 to 1,530 employees in 2013. 3.7 Preconstruction Learning Curve One of the obvious trends shown by the screening sur- vey and the agency interviews is the abundance of data that exists within a typical agency and the lack of reliable tools to organize and convert the data into actionable information. In all cases, agency engineers were required to bill their time to specific projects, making the information available within the agency’s financial accounting system. On some occasions, staff did not think that the hours billed were completely accurate. Oklahoma DOT responded that approximately 50% of its time was billed to departmental overhead instead of to a spe- cific job. This noise in the data is to be expected when working in this environment; the important thing to recognize when using these data is that the noise is there, so try to understand how it will affect the outcome of the estimate. In agencies with a high level of noise, a top-down approach would work better than a bottom-up estimate because of the lack of precise his- toric labor-hour information. For this research, there was no choice and noisy data had to be used. Therefore, the concept of developing two or more ways to estimate the same project is extremely important to the accuracy of the PCS cost estimate. The proposed method is documented in a simple and adapt- able manner in the guidebook, and the parametric estimating model has been developed so that it can be adopted by an agency and adjusted to fit its needs. With the proposed system in place, DOTs will no longer be as reliant on professional judgment for PCS cost estimat- ing. This becomes more critical as DOTs are downsized. The PCS cost-estimating system is a knowledge management tool that institutionalizes the process, making it more consistent as DOTs lose experienced mid-career staff to the private sec- tor. Feedback during the interviews shows that any system needs to be simple and user-friendly. This also demonstrates Agency Model Use Arizona Third-party transaction cost– benefit analysis Unaware Louisiana Outsourcing decision assistance model (2002) available on CD- ROM by request No Oregon Decision tree – cost-based outsourcing decisions (2007) Could not confirm Table 3.12. Outsourcing cost comparison models (Ismail and Sutliff 2011).

33 the need for PCS training within DOTs on the importance of estimating PCSs as well as on effective practices to help improve PCS estimates. 3.8 Case Study Summary 3.8.1 Possible Effective Practices A number of PCS cost-estimating methods have been iden- tified throughout the case study analysis. A list of possible effective practices has been compiled. These were reviewed in Phase 2 of the research. • Back-check of hours. This method was used by UDOT to check the estimate using a direct estimate of hours against a past project cost range. This is a useful tool to validate the estimate, especially for younger estimators with less experience. • Life-cycle project manager. This practice assigns a project manager for the entire duration of the project’s develop- ment and delivery period. This allows the project manager to control the costs and review the estimate as issues arise or the scope changes. • Estimate check milestones. This process has been imple- mented by UDOT and is demonstrated in Figure 3.5. The practice ensures that estimates are reviewed and updated at various stages of the project development process. 3.8.2 Data Quality Once the research team began developing databases from the data collected, reduced, cleaned, and collated, it found it difficult to be able to guarantee that the data were com- plete and accurate. A number of inconsistencies have been observed within collected data sets, and numerous blank cells indicated that some information was not complete. These issues of data quality and data availability created a number of challenges for the research team. The outputs of any model are only as good as its input, placing the quality of any esti- mating model produced into question. After becoming familiar with the Utah and Iowa data sets, the team became concerned at how different the two sets were. Such variation between the agencies’ classification of data indicated that it would be difficult to achieve a consistent result between the two DOT’s models. These observations were reported to the panel in the Octo- ber 2014 quarterly report. The panel was notified that the research might ultimately lead to a process model that must be customized specifically to the way data are collected by a specific DOT rather than a fairly generalized approach that can be used by all DOTs. The research team also noted at this point that implement- ing the research might be more difficult than expected as it would require a significant investment by each agency to con- figure the data it currently maintains.