Evaluation of Project Delivery Methods (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

TCRP G-08 – Project Delivery Methods Chapter 9 - PDM Decision Tool Evaluation CHAPTER 9 – EVALUATION OF PROJECT DELIVERY METHOD DECISION TOOL The structured interviews and case studies followed the methodology of Yin (2004) and Oppenheim (1992). According to Yin, case studies are appropriate to use for exploratory or explanatory questions, such as what project characteristics can be matched with project delivery method advantages to make an effective project delivery method selection decision for major transit projects? Case studies are also appropriate when the researcher does not require control over the events and when the research focuses on contemporary events. Both of these conditions apply to the study. The unit of analysis or the “case” is the project, starting with the project conceptual and systems planning and ending at the conclusion of construction. To assure the quality of the case study design and evaluation, the following measures recommended by Yin were included: • Construct validity: The team submitted the case study data collection and interview guide to the TCRP panel to establish that correct operational measures were used. This study included multiple sources of evidence by interviewing more than one expert in each transit agency, as well as having the experts review the key information included in the Interim Report. Additionally, a pilot study was conducted initially to refine data collection procedures. • Internal validity: to establish a causal relationship between the critical factors and the strategies and techniques, this study applied the hypotheses to a number of cases that include 1) a range of project types, settings, and conditions, 2) a range of different project delivery methods, 3) a range of project performance outcomes (both positive and negative). Additionally, a draft of the Tier 1 and 2 decision tool was applied by a research team member to a project that was not included in the case study data collection to determine two factors. First, it demonstrated that the decision tool could be applied outside the universe formed by the case study projects and secondly, it confirmed the utility of the tool by comparing the PDM selected by the tool with the PDM that was favored by the agency at the time of the internal validation effort. • External validity: a validation workshop was conducted at the 2007 DBIA Transportation conference Minneapolis and was used to confirm the findings from the case studies. Input from the workshop helped to focus the research on the four delivery methods covered by the decision tool. Once the tool had been developed it was given to four case study agencies who were asked to run through the process and if possible, compare the results with one of their projects. • Reliability: to demonstrate that the study procedures and results were reliable and repeatable, this study prepared a detailed case study protocol. The research protocol Northeastern University The Research Report 169

TCRP G-08 – Project Delivery Methods Chapter 9 - PDM Decision Tool Evaluation followed the methods that were reviewed and approved by the TCRP panel in the final work plan. Final Validation of PDM Multi-Tier Decision Tool The above validation discussion shows how the research team was careful to validate its work both before and during the study. This effort was valuable and paid dividends in the final validation efforts. Figure 9.1 demonstrates the process that was followed to achieve final internal and external validation. Figure 9-1 - Project Delivery Decision Tool Final Internal and External Validation Flow Chart Northeastern University The Research Report 170

TCRP G-08 – Project Delivery Methods Chapter 9 - PDM Decision Tool Evaluation In essence, the process was to first apply Tiers 1 and 2 to an actual project by a member of the research team to identify if the tool was working as designed and identify the gaps in the process that needed to be filled. Once those results were synthesized and the two tiers were revised, the tool was distributed to four of the case study project agencies. They were asked to apply the tool to a project that they felt had been successfully delivered using the appropriate PDM. They were then asked to compare the resulting PDM from the tool matched the PDM that had been used on the test project. The tool is deemed to be valid if the internal and external parties determine that the tool led them to the logical PDM for the project characteristics that were considered. Finally, it should be noted that due to the relative rarity of DBOM projects, it was not possible to specifically validate the tool for that PDM. However, the DB validation in fact covers the initial capital delivery phase of these type projects and therefore, DBOM can be partially validated by the DB validation effort. Final Internal Validation As previously stated, a member of the research team applied the initial draft of the Tier 1 and 2 PDM decision tool to an upcoming project for a large metropolitan transit agency. The tool yielded a strong indication for DB project delivery. The effort was fruitful in that it pointed out some minor weakness in the Tier 1 logic and furnished constructive criticism on Tier 2. The major issue was the need for a multi-disciplinary group to collaboratively apply the PDM tool to ensure that critical aspects of design, construction, and operations are all adequately represented in consensus ratings of project issues. Final External Validation As previously discussed, the Tier 1 and 2 decision tool was furnished to four of the case study agencies. One of those agencies requested that their identity be not disclosed. Therefore, as the sample of case study agencies is small, no agencies will be identified in the subsequent paragraphs to comply with that reasonable request and prevent inadvertently identifying the agency by process of elimination. Each agency was asked to apply the tool to a specific project that was either upcoming or had been completed. The purpose for this suggestion was that it was the team’s belief that the selection system works best when the user can carefully define project objectives and goals. It was felt that if the agencies had a specific project in mind they would be in a position to better define project characteristics and goals. Additionally, each agency was asked to rate the effectiveness of the tool in the following categories (see Appendix E for a copy of the questionnaire and rating form): • Comprehensiveness • Clarity • Applicability to real projects • Contribution to resulting in a transparent and defensible decision • Overall satisfaction In all four cases, the tool was deemed to be valid. No unsatisfactory ratings were recorded. The specific adjectival evaluations in the specific categories are as follows: Northeastern University The Research Report 171

TCRP G-08 – Project Delivery Methods Chapter 9 - PDM Decision Tool Evaluation Northeastern University The Research Report 172 • Comprehensiveness: 3 excellent; 1 very good • Clarity: 4 very good • Applicability to real projects: 3 excellent; 1 very good • Contribution to decision: 2 excellent; 1 very good; 1 good • Overall satisfaction: 3 excellent; 1 good Specific comments were primarily directed at the Tier 1 process. Most users found it to be comprehensive. One agency indicated that it might include more detail in the political influence issue portion. Another thought that it could be broken into a first part that would apply to all projects and a second part that would be project-specific to streamline the use over time by the same agency. Finally, two of the agencies indicated that the tool would be particularly valuable to an agency that was facing the use of alternative project delivery for the first time. One agency had this to say about that issue: “I think that this would be particularly helpful to an agency using alternate delivery for the first time, but more experienced agencies can benefit from the comprehensive list of items to be considered.” Overall comments regarding the validity of the tool from each of the four agencies are as follows: • Agency with DBB, CMR, and DB experience: “Seems like a fine tool. The list of topics and advantages/disadvantages are very complete and thorough. The discipline of the Tier 2 analysis is a good way to stress the most important factors for the job under study.” • Agency with DBB, CMR, and DB experience: “It is an excellent document and should be very helpful to those in the project development of major transit projects.” • Agency with DBB and DB experience: “Very interesting approach.” • Agency with DBB, DB, and DBOM experience: “At first the process looks a bit long but I think as people use it they will find it helpful.” Thus, the Tier 1 and 2 PDM decision tool framework is found to be valid in accordance with the procedures established by the research team and approved by the TCRP oversight panel. Another finding of the validation process was that in all cases, there remained no ambiguity on the choice of PDM after going through the first two tiers. Because of this, the validation and testing of Tier 3 could not be accomplished in the same way.