Evaluation of Project Delivery Methods (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach CHAPTER 7 – TIER 2 – WEIGHTED-MATRIX DELIVERY DECISION APPROACH Introduction The Tier 2 Weighted-Matrix Delivery Decision Approach provides a means for owners to further examine and document a project delivery decision for an individual project. In the case that an obvious choice was not found in the Tier 1 Analytical Delivery Decision Approach, the Tier 2 approach provides owners with a process to select a delivery method by prioritizing project objectives and selecting the delivery method that best aligns with these objectives. The Tier 2 Weighted-Matrix Delivery Approach is founded upon successful delivery decision tools developed by academics and professionals over the past 20 years (Loulakis 2000; CII 2003; Skitmore & Marsden 1988). Owners should complete a Tier 1 review before conducting a Tier 2 review. The Tier 1 review provides owners with two key pieces of information. First, Tier 1 requires owners to define their project goals in terms of cost, schedule, quality, maintainability, sustainability, and other options. These project goals are critical to the Tier 2 review. Second, Tier 1 provides a short list of available project delivery options. Only those project delivery methods that are feasible and which have the best potential for a successful application will pass through the Tier 1 filtering process. The filtering process involves an examination of go/no-go issues and also an examination of 24 critical issues involved in the project delivery decision. Knowledge of these critical issues will be helpful in the Tier 2 decision-making process. The Tier 2 approach has three primary objectives: • Present a structured framework to assist agencies in prioritizing their unique project goals and delivery selection issues; • Assist owners in aligning their unique goals and issues with the most appropriate project delivery method; and • Further document the project delivery decision in the Project Delivery Decision Report established in Tier 1. The Tier 2 approach provides a framework for agencies to prioritize their project goals and select the project delivery method that best aligns with these goals. The motivation for this approach is to capture the fact that priorities for project goals and critical selection issues are unique to each project. Likewise, the delivery methods vary in their ability to achieve these goals and deal with these issues. The Tier 2 approach will align these two facets of the delivery decision. Northeastern University The Research Report 150

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach At the completion of Tier 2, there is still a possibility that the agency may not have a single, clear and logical choice for a project delivery method. If this is the case, the agency will be advised to move to the Tier 3 selection processes with the best delivery method options and make the final decision based upon a detailed risk analysis of the issues involved with each delivery method. The Tier 2 approach is comprised of five distinct steps listed below and shown in Figure 7.1. Step 1. Define Selection Factors Step 2. Weight Selection Factors Step 3. Score Project Delivery Methods Step 4. Choose Most Appropriate Project Delivery Method Step 5. Document Results Step 1. Define Selection Factors Time: _______________ Cost: _______________ Quality: _______________ … : _______________ … : _______________ … Step 5. Document Results PROJECT DELIVERY DECISION REPORT Tier 1 • Project Description • Project Goals • Delivery Methods Considered • Advantages and Disadvantages Tier 2 • Weighted-Matrix Decision Chart Step 2. Weight Selection Factors Step 3. Score Project Delivery Methods Step 4. Choose Most Appropriate Project Delivery Decision PROJECT DELIVERY METHOD DBB CMR DB DBOM Selection Factor Factor Weight Score Weighted Score Score Weighted Score Score Weighted Score Score Weighted Score Factor 1 (e.g., Project Goals) Factor 2 (e.g., Agency experience) Factor 3 (e.g., Market issues) Factor 4 to 7 … Total Score Figure 7-1 - Tier 2 Selection Process Overview Step 1 of the Tier 2 process begins by defining a concise set of selection factors. These selection factors consist of the project goals and any additional critical issues examined in Tier 1. The Tier 1 process asks owners to establish their project goals at the very beginning of the process. The first step in Tier 2 asks the owners to develop a concise set of selection factors by combining their project goals and with the most important of the 24 critical issues examined in Tier 1. The Tier 2 method will use these selection factors throughout the process. Step 2 asks owners to rank and then weight selection factors. The project goals may overlap with the critical issues, in which case they can be combined. Other critical issues will stand alone for analysis. Step 2 will result in a list of up to seven project goals and critical issues for further analysis. Northeastern University The Research Report 151

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach Step 3 of the Tier 2 process requires owners to score each delivery method in terms of the selection factors. A further examination of the advantages and disadvantages for each delivery methods will form the basis for these scores. Since the scores will be subjective, the owners will need to be diligent in documenting the reasons for the scores. Step 4 involves a determination of the most appropriate delivery method through the completion of the weighted decision matrix. Owners will make the determination by multiplying the selection factor weights by the project delivery scores and then summing the values. The delivery method with the highest score will indicate the best choice. However, since the scores will be subjective, the owners will be encouraged to review the totals to determine if the values are logical and defensible on the basis of their professional judgment. The objective of Step 5 is to supplement the documentation of the Project Delivery Decision Report developed in Tier 1. The Tier 1 report will provide project description, project goals, delivery methods considered, advantages and disadvantages, delivery method decision, and any relevant appendices. The Tier 2 documentation will include a documentation of the weighted decision matrix to supplement the archival record for the project delivery decision. It will serve to communicate the decision to interested stakeholders and to justify the decision if issues arise years later as the project is completed. Step 1. Define Selection Factors As stated in Step 1 of Tier 1, understanding and communicating a concise set of project goals is perhaps the most important element in selecting an appropriate project delivery method. The definition of project goals is a key success factor in not only the project delivery decision, but also the development of procurement documents and the administration of a project. It is the performance goals (e.g., time, cost, quality, maintainability, and sustainability) that typically drive the project delivery decision. The first step in Tier 2 requires owners to combine the project goals and critical issues into a set of selection factors for use in the weighted decision matrix. This step requires a review and filtering of the project goals and critical issues for use as selection factors. Figure 7.2 depicts this process. Northeastern University The Research Report 152

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach Figure 7-2 - Tier 2 Selection Factor Development To create the goal-based selection factors, owners should review the project goals that were established in Tier 1. The Tier 1 review of the delivery method advantages and disadvantages may have revealed overlaps or gaps in the originally established project goals. While the original project goals should not change, these overlaps and gaps will need to be removed for the development of section factors. Step 1 in the Tier 2 decision process invites edits to these goals as they are rewritten into selection factors. In developing the selection factors from the project goals, owners should consider the following questions: • Are there significant overlaps in the project goal statements that can be revised to make them more independent? • Are there missing goal statements that are needed to define the ultimate project success? • Can any of the goals be stated more concisely? The Tier 1 process provided an opportunity to review 24 critical project delivery issues. However, the Tier 1 process treated all of the issues as equally important. Upon reviewing the issues, owners will certainly find that all of the issues are not of equal importance. A small number of these issues will likely be critical to the final project delivery decision. The next task in Step 1 is to select up to seven of the most critical issues to examine to develop as selection factors. The owner should select the most critical issues based on the following criteria: • The critical issue should be independent of the project goals; • The critical issues should be independent of each other; and • No more than seven critical issues should be chosen. The final task of Step 1 is to provide a consolidated list of the goals-base and issue-based selection factors into one comprehensive list. The next step in the Tier 2 process involves a ranking of the goals and critical issues; therefore one combined list is required. Figure 7.3 provides and example listing of selection factors for a hypothetical project. This example will be used throughout the Tier 2 process description that follows. Northeastern University The Research Report 153

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach The list of selection factors in this example illustrates a concise set of criteria that can be used for selecting the appropriate project delivery method. The list below shows examples of project goals relating to time, cost and sustainability and a critical issue regarding the agency staffing. While other issues of technical quality, maintainability, third party agreements, etc., undoubtedly exist on the project, the list below constitutes the primary goals and issues that will measure the success of the project at its completion and can thereby be used as selection factors. Project Name: Example Project • Project complete by November 1, 20XX. • Cost not to exceed $1.5 billion. • Enhance the environment through less traffic congestion and pollution. • Minimize staffing requirements during design and construction. Figure 7-3 – Example Listing of Selection Factors Step 2. Weight Selection Factors The Tier 2 process is based on the premise that owners can establish a unique hierarchy of selection factors. In other words, each project will define success differently and the criteria for success can be described by a few key selection factors. The objective of Step 2 is to weight the list of selection factors. Step 2 involves a process of first ranking and then weighting the selection factors. There are numerous methods to achieve a weighted ranking of the factors. The most straight-forward method is a direct ranking and weighting through a discussion and consensus-building meeting with project decision makers. The decision will by nature be somewhat subjective, so a group decision with diligent documentation should be applied. To achieve the weighted ranking, owners should apply the following steps: a. List the selection factors in rank order from the highest to the lowest bearing project success. b. Include a minimum of four (4) and a maximum of seven (7) factors. o Remove factors not ranked in the top seven (7). c. Using 100 total points, weight the factors according to their influence on project success. o Avoid equal weighting of factors. o Remove any factors with a value of less than five (5) of the 100 points and redistribute points. Northeastern University The Research Report 154

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach These three steps describe a simple method for achieving a weighted ranking of the selection factors. Decision sciences provide more precise methods for achieving weighted rankings and developing a consensus. The result of Step 2 will be a weighted ranking of up to seven (7) selection factors. The weightings should total 100 points. Equal factor weightings are not recommended because distinguishing the importance between factors (goals and critical issues) is necessary for the decision process. Additionally no single factor should have a point value of less than five (5) because it will not have an influence on the final decision and may in fact make the selection more difficult. The next steps will involve combining the weighted rankings with a scoring of the project delivery methods to arrive at a final selection of the most appropriate delivery method. Figure 7.4 continues the previous example by providing weighted rankings for factors. The list below shows examples of project goals and issues that have been weighted to reflect their influence on success for the given project. These weightings are project dependent and should be agreed upon by key owner team members. Project Name: Example Project Weight Goal/Issue 50 Project complete by November 1, 20XX. 25 Cost not to exceed $1.5 billion. 15 Enhance the environment through less traffic congestion and pollution. 10 Minimize staffing requirements during design and construction. 100 Total Figure 7-4 - Example of Weighted Ranking for Selection Factors Step 3. Score Project Delivery Methods The third step involves a scoring of the alternative delivery methods from the Tier 1 analysis. Each of these delivery methods will have a bearing or influence on the selection factors, which stem from the project goals and critical issues. The key decision makers must translate this influence into a score to arrive at a decision. To achieve the total scores for each delivery method, owners should apply the following steps: a. Using the scale in Table 7.1, assign a score to each delivery method as they relate to the selection factor. Score all delivery methods for each factor before moving to the next factor. b. Repeat step “a” for each for each selection factor. c. When all delivery methods have been scored, multiply the factor weight by the score to achieve a weighted score for each delivery method. d. Sum all of the weighted scores to arrive at a total score for each delivery method. Table 7.1 provides a scale for scoring each delivery method as they relate to the selection factor. The scores range from 1 to 10 so that, when they are multiplied by the factor weight, the total score Northeastern University The Research Report 155

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach will range from 0 to 1000. The scores are subjective, so a detailed definition for each is provided adjacent to the score. When scoring the delivery methods, owners should discuss the advantages and disadvantages of each delivery method. The alignment of these advantages and disadvantages with the selection factors forms the basis for the scoring. In assigning the scores, the owner should work in a team, build a consensus decision, and carefully document the reasons for choosing each individual score. Consideration should be given to the relative scores for each delivery method to ensure consistency. Table 7-1 - Project Delivery Scoring Scale (adapted from Saaty 1990) SCORE DEFINITION 10 The evidence that the delivery method positively aligns with the project objective or issue is of the highest possible order of affirmation. 8 The delivery method strongly aligns with the objective or issue and is demonstrated in practice. There is a slight risk that the objective or issue may not be beneficial. 6 Experience and judgment point to the delivery method strongly aligning with the objective or issue. There is a mild risk that the objective may not be beneficial. 4 Experience and judgment slightly points to the delivery method aligning with the objective or issue. There is a strong risk that the objective will be negatively affected. 2 There is little benefit to applying the delivery method for this goal or objective. There is a strong likelihood that the object will not be achieved. 9,7,5,3,1 Intermediate values between two adjacent judgments. Similarly to the development of factor weights, the scoring can be done simply through a group discussion among key owner team decision makers. Table 7.2 provides a weighted decision matrix template. The matrix can contain up to four (4) delivery methods, depending upon the results of Tier 1. The matrix can also contain up to seven (7) selection factors for each project. Northeastern University The Research Report 156

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach Table 7-2 - Weighted-Matrix Template PROJECT DELIVERY METHOD DBB CMR DB DBOM Selection Factor Factor Weight Score Weighted Score Score Weighted Score Score Weighted Score Score Weighted Score Factor 1 (e.g., Project Goals) Factor 2 (e.g., Agency experience) Factor 3 (e.g., Market issues) Factor 4 to 7 … Total Score The result of Step 3 will be a scored ranking of the delivery methods in question. The delivery method with the highest total score will be the most appropriate method for the given project. The next steps involve documenting the individual scores and creating a Project Delivery Selection Report. Figure 7.5 continues the previous example by scoring the delivery methods as they relate to each selection factor. A very brief documentation for the scoring follows the table. Northeastern University The Research Report 157

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach The table below provides an example of how an owner might score the project delivery methods for a particular project. Note that only the CMR and DB project delivery methods made it through the Tier 1 filter for further consideration in Tier 2. Also note that the scores in the example below are project dependent and they will certainly change from project to project. Example Project Decision Matrix PROJECT DELIVERY METHOD CMR DB Selection Factors Factor Weight Score Weighted Score Score Weighted Score Project complete by November 1, 20XX 50 6 300 8 400 Cost not to exceed $1.5 billion 25 6 150 8 200 Enhance the environment through less traffic congestion and pollution 15 10 150 6 90 Minimize staffing requirements during design and construction 10 8 80 6 60 Total Score 100 680 750 Explanation of Scores Project completion factor: The project completion factor relates to a project goal. In this case the project has a fixed end date of November 1, 20XX. The owner believes that CMR delivery can achieve the completion date. The owner also believes that CMR will require the use of multiple bid packages to achieve the schedule, which adds a risk for meeting the schedule date (CMR = 6). DB delivery provides for a single entity to coordinate design and construction. DB also allows for an owner to specify a fixed end date in the procurement documents and the contract. The owner is confident by what has been demonstrated in practice that the end date can be achieved through a DB delivery (DB = 8). Cost containment factor: The cost containment factor relates to a project goal. The project has a maximum budget of $1.5 billion. DB delivery has demonstrated in practice that a fixed price can be set early in the project development process and it has also been demonstrated that DB provides the lowest average cost growth of the two methods in question (DB = 8 in this case). CMR also provides the ability to meet a fixed price, but the owner is not as confident with the experience using a guaranteed maximum price contract structure. They also feel that there is more risk of not achieving the schedule with CMR when compared to DB (CMR = 6 in this case). Northeastern University The Research Report 158

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach Sustainability factor: The sustainability factor relates to a project goal. The owner desires that the project will enhance the environment through less traffic congestion and pollution. In this case, the owner has met with designers who can help define sustainability goals that can be achieved through their independent designs if they are hired directly by the owner. The CMR delivery method will provide for a direct contract between the owner and the designer to ensure that the goals are achieved (CMR = 10). While the owner can develop DB performance criteria related to sustainability, they are not as confident that they can accurately articulate their goals in the performance criteria and they believe there is a risk that the goals will not be fully achieved (DB = 6). Agency requirements factor: The agency requirement factor relates to a critical issue examined in Tier 1. The owner does not have a large staff and desires to minimize staffing requirements during design and construction. The CMR option will allow the owner to supplement their staff during both design and construction, either with the designer or with the CMR. The owner is confident that qualified professionals exist to meet their staffing needs, but is slightly concerned about exactly how the working relationship will be executed between the CMR and the owner (CMR = 8). The DB option will require owner to mass its resources (or build up for a short time) during the procurement and design review process. The owner believes that they can supplement their staff with a general engineering consultant, but they are not confident that the DB option will be as effective as the CMR option (DB = 6). Figure 7-5 - Example of Weighted Ranking for Project Goals and Critical Issues Step 4. Choose the Most Appropriate Project Delivery Method At this point, choosing the appropriate delivery method is simply a matter of reviewing the total scores and making the project delivery decision. Since the factor weighting and the scores are subjective, the owner should review the totals and confirm that they are logical and defensible. If upon further discussion, a factor weight or project delivery score appears to be incorrect or overly influence the selection, it is acceptable to make changes and create a new total project score. The key is to document the reasons for each change. If the owner is not confident in a particular weight or score, they can conduct more research about a particular delivery method and revisit the scoring after gathering more information. If at this point, a single project delivery decision is not apparent, the owner should document the results and move to the Tier 3 decision process (Chapter 8). Step 5. Document Results As in Tier 1, documentation of the delivery decision is a key portion of the process. Whether one delivery method clearly achieves the highest score or no dominant choice appears, documentation is a vital step. Documentation will assist in developing procurement and contracting strategies for the ultimate project delivery method. It will also serve to communicate the project delivery choice to interested stakeholders. Northeastern University The Research Report 159

TCRP G-08 – Project Delivery Methods Chapter 7 - Weighted Matrix Approach Northeastern University The Research Report 160 Documentation of Tier 2 involves supplementing the Project Delivery Decision Report developed in Tier 1. The Project Delivery Decision Report should contain the weighted matrix and also a detailed documentation of the reasoning that was used to assign each criterion weight and project delivery score. Conclusions The Tier 2 Weighted-Matrix Delivery Decision Approach extends the structured Tier 1 approach through an examination of how project delivery methods align with project goals and critical issues as defined through selection factors. The weighted ranking of project selection factors requires decision makers to examine their priorities and make a closer examination of the attributes for each delivery that passed the Tier 1 filter. At the end of Step 4, there may be a single, clear and logical choice for a project delivery method and the choice can be documented through a Project Delivery Decision Report. If a dominant choice does not appear, the agency should document the results and move to the Tier 3 selection process to examine how the delivery methods relate to the project risks.