A Comprehensive Development Plan for a Multimodal Noise and Emissions Model (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

G-1 APPENDIX G. ROUND 1 EVALUATION – RATINGS, SCORES, AND STATISTICS Twenty members of ACRP 02-09 Panel and the project team evaluated 5 model design concepts based on the modified Pugh Matrix. Tables G-1 to G-6 present the performance criteria ratings. The identities of the evaluators are anonymous as they are not pertinent to the scoring. Nonparametric statistics, median, and quartiles, are provided below the individual ratings. Nonparametric statistics are used instead of typical (parametric) statistics, like mean and standard deviation. The reason is that there is not enough information to determine whether the evaluators’ ratings conform to a normal distribution; a necessary requirement for use of parametric statistics. The median is the middle value of an ordered set of values. Quartiles divide an ordered distribution into four parts each containing one quarter of the scores. The 1st quartile is the point in a given distribution at which 25% of the observations fall below that point and 75% of the observations fall above it. The 3rd Table G-7 contains the ratings for the cost implications criterion and is identical in structure to the previous performance rating tables. The horizontal axis of the cost implications statistic chart is reversed from the previous performance charts to denote that the best rating is 1 (lowest cost implications) and 5 is the worst (highest cost implications). quartile is the point in a given distribution at which 75% of the observations fall below that point and 25% of the observations fall above it. The statistics are also shown graphically with a legend. Since the ratings are ordinal values (1, 2, 3, 4, and 5), it is possible that the median, first, and/or third quartile might be the same. Table G-1 (Agency Acceptance) contains 2 examples of this for Alternatives #3 and 4. Table G-8 contains the performance score (P), which is the weighted sum of the evaluation scores according to the following equation i i i RwP ⋅=∑ = 6 1 Where; Ri w is the performance criterion rating i Table G-9 contains the cost score (C), which is identical to Table G-7 because the cost implications weight is 1. is the performance criterion weighting Table G-10 contains the value score (P/C). Alternative #1 (Build on AEDT) received the highest median value score of 1.31 making it the winner in Round 1. Figure G-1 combines the rating statistics charts for the 6 performance criteria and one cost criterion. Figure G-2 combines the statistics charts for the performance, cost, and value scores.

G-2 TABLE G-1 Round 1 Scores - Agency Acceptance Performance Evaluation Criteria Weight Agency Acceptance 0.30 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 3 3 2 2 Evaluator 2 3 5 2 3 3 Evaluator 3 3 3 3 3 4 Evaluator 4 3 3 2 1 1 Evaluator 5 3 2 1 1 2 Evaluator 6 3 4 3 2 2 Evaluator 7 3 2 4 1 2 Evaluator 8 3 5 2 1 2 Evaluator 9 3 2 1 2 2 Evaluator 10 3 2 2 2 2 Evaluator 11 3 2 2 2 2 Evaluator 12 3 3 2 1 1 Evaluator 13 3 1 2 1 2 Evaluator 14 3 5 5 2 2 Evaluator 15 3 3 2 2 3 Evaluator 16 3 2 2 2 2 Evaluator 17 3 2 2 1 2 Evaluator 18 3 2 3 2 2 Evaluator 19 3 4 3 1 2 Evaluator 20 3 2 2 2 2 3.00 2.50 2.00 2.00 2.00 3.00 2.00 2.00 1.00 2.00 3.00 3.25 3.00 2.00 2.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e Median 3rd Quartile 1st Quartile

G-3 TABLE G-2 Round 1 Scores - Technical Feasibility Performance Evaluation Criteria Weight Technical Feasibility 0.20 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 4 2 3 3 Evaluator 2 3 5 3 4 4 Evaluator 3 3 3 4 5 4 Evaluator 4 3 3 3 2 4 Evaluator 5 3 2 2 2 2 Evaluator 6 3 4 2 3 2 Evaluator 7 3 2 1 4 2 Evaluator 8 3 3 4 3 3 Evaluator 9 3 3 1 3 2 Evaluator 10 3 4 3 3 3 Evaluator 11 3 4 2 2 3 Evaluator 12 3 3 3 3 2 Evaluator 13 3 3 1 1 1 Evaluator 14 3 5 3 4 1 Evaluator 15 3 4 3 4 3 Evaluator 16 3 5 2 3 3 Evaluator 17 3 3 3 3 3 Evaluator 18 3 3 2 2 2 Evaluator 19 3 5 4 4 3 Evaluator 20 3 4 2 2 2 3.00 3.50 2.50 3.00 3.00 3.00 3.00 2.00 2.00 2.00 3.00 4.00 3.00 4.00 3.00 Design Concepts Median 1st Quartile 3rd Quartile Evaluator D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e Median 3rd Quartile 1st Quartile

G-4 TABLE G-3 Round 1 Scores - Analytical Proficiency Performance Evaluation Criteria Weight Analytical Proficiency 0.20 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 3 3 3 2 Evaluator 2 3 3 3 4 4 Evaluator 3 3 1 3 4 3 Evaluator 4 3 1 3 3 3 Evaluator 5 3 2 4 3 4 Evaluator 6 3 2 3 2 3 Evaluator 7 3 2 4 4 2 Evaluator 8 3 4 2 3 3 Evaluator 9 3 3 4 4 4 Evaluator 10 3 2 3 3 3 Evaluator 11 3 2 3 3 3 Evaluator 12 3 3 3 2 2 Evaluator 13 3 1 2 4 1 Evaluator 14 3 4 3 3 3 Evaluator 15 3 3 3 4 3 Evaluator 16 3 2 2 4 3 Evaluator 17 3 2 3 3 2 Evaluator 18 3 3 2 3 3 Evaluator 19 3 3 2 4 2 Evaluator 20 3 2 4 2 2 3.00 2.00 3.00 3.00 3.00 3.00 2.00 2.75 3.00 2.00 3.00 3.00 3.00 4.00 3.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00Rating Al te rn at iv e Median 3rd Quartile 1st Quartile

G-5 TABLE G-4 Round 1 Scores - Scalability Performance Evaluation Criteria Weight Scalability 0.15 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 4 3 3 3 Evaluator 2 3 4 3 3 4 Evaluator 3 3 2 2 3 5 Evaluator 4 3 2 4 4 4 Evaluator 5 3 3 4 3 4 Evaluator 6 3 3 4 3 2 Evaluator 7 3 3 4 4 2 Evaluator 8 3 2 4 2 2 Evaluator 9 3 4 4 4 4 Evaluator 10 3 4 4 4 4 Evaluator 11 3 3 3 4 4 Evaluator 12 3 3 3 3 3 Evaluator 13 3 3 4 4 2 Evaluator 14 3 5 3 5 4 Evaluator 15 3 4 3 4 3 Evaluator 16 3 4 4 4 4 Evaluator 17 3 2 3 3 3 Evaluator 18 3 2 3 3 3 Evaluator 19 3 4 5 3 4 Evaluator 20 3 3 4 4 2 3.00 3.00 4.00 3.50 3.50 3.00 2.75 3.00 3.00 2.75 3.00 4.00 4.00 4.00 4.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e Median 3rd Quartile 1st Quartile

G-6 TABLE G-5 Round 1 Scores - Responsiveness Performance Evaluation Criteria Weight Responsiveness 0.10 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 4 2 2 1 Evaluator 2 3 3 2 3 3 Evaluator 3 3 2 2 4 3 Evaluator 4 3 2 2 2 3 Evaluator 5 3 2 2 4 2 Evaluator 6 3 3 4 3 3 Evaluator 7 3 3 4 4 3 Evaluator 8 3 2 3 2 2 Evaluator 9 3 2 3 2 3 Evaluator 10 3 3 3 2 3 Evaluator 11 3 2 3 1 3 Evaluator 12 3 2 3 1 1 Evaluator 13 3 3 4 2 3 Evaluator 14 3 4 3 1 3 Evaluator 15 3 3 2 2 3 Evaluator 16 3 3 4 4 4 Evaluator 17 3 2 2 1 2 Evaluator 18 3 3 2 2 3 Evaluator 19 3 2 3 3 3 Evaluator 20 3 2 4 2 2 3.00 2.50 3.00 2.00 3.00 3.00 2.00 2.00 2.00 2.00 3.00 3.00 3.25 3.00 3.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e Median 3rd Quartile 1st Quartile

G-7 TABLE G-6 Round 1 Scores - International Credibility Performance Evaluation Criteria Weight International Credibility 0.05 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 3 4 3 4 Evaluator 2 3 4 3 4 5 Evaluator 3 3 1 4 5 5 Evaluator 4 3 3 4 4 4 Evaluator 5 3 3 4 5 4 Evaluator 6 3 4 2 3 2 Evaluator 7 3 3 3 3 3 Evaluator 8 3 4 2 1 4 Evaluator 9 3 2 3 3 4 Evaluator 10 3 3 3 3 3 Evaluator 11 3 4 3 4 4 Evaluator 12 3 3 3 1 2 Evaluator 13 3 2 3 3 3 Evaluator 14 3 4 3 5 5 Evaluator 15 3 3 3 4 4 Evaluator 16 3 4 3 2 4 Evaluator 17 3 2 3 2 4 Evaluator 18 3 4 4 3 3 Evaluator 19 3 4 3 3 4 Evaluator 20 3 3 4 4 5 3.00 3.00 3.00 3.00 4.00 3.00 3.00 3.00 3.00 3.00 3.00 4.00 4.00 4.00 4.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e Median 3rd Quartile 1st Quartile

G-8 TABLE G-7 Round 1 Scores - Cost Implications Performance Evaluation Criteria Weight Cost Implications 1.00 Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 2 4 3 4 Evaluator 2 3 2 3 2 2 Evaluator 3 3 2 3 3 2 Evaluator 4 3 2 4 4 4 Evaluator 5 3 4 3 3 3 Evaluator 6 3 1 2 5 4 Evaluator 7 3 4 5 5 5 Evaluator 8 3 2 3 5 3 Evaluator 9 3 2 4 4 4 Evaluator 10 3 2 4 4 4 Evaluator 11 3 2 4 4 3 Evaluator 12 3 4 4 5 4 Evaluator 13 3 3 4 4 3 Evaluator 14 3 1 2 3 1 Evaluator 15 3 2 4 5 3 Evaluator 16 3 2 2 4 5 Evaluator 17 3 2 3 4 4 Evaluator 18 3 4 2 4 5 Evaluator 19 3 3 3 4 3 Evaluator 20 3 2 5 4 5 3.00 2.00 3.50 4.00 4.00 3.00 2.00 3.00 3.75 3.00 3.00 3.00 4.00 4.25 4.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.001.502.002.503.003.504.004.505.00 Rating A lte rn at iv e Note that Rating scale is reversed. Median 3rd Quartile 1st Quartile

G-9 TABLE G-8 Round 1 Scores - Performance Score Performance Score Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3.00 3.45 2.75 2.60 2.35 Evaluator 2 3.00 4.20 2.60 3.45 3.65 Evaluator 3 3.00 2.25 3.00 3.80 3.90 Evaluator 4 3.00 2.35 2.80 2.30 2.80 Evaluator 5 3.00 2.20 2.50 2.40 2.80 Evaluator 6 3.00 3.35 3.00 2.50 2.30 Evaluator 7 3.00 2.30 3.35 3.05 2.15 Evaluator 8 3.00 3.60 2.80 2.05 2.50 Evaluator 9 3.00 2.70 2.35 2.95 2.90 Evaluator 10 3.00 2.85 2.85 2.75 2.85 Evaluator 11 3.00 2.65 2.50 2.50 2.90 Evaluator 12 3.00 2.90 2.70 1.90 1.75 Evaluator 13 3.00 1.95 2.35 2.25 1.75 Evaluator 14 3.00 4.65 3.60 3.10 2.55 Evaluator 15 3.00 3.35 2.60 3.20 3.05 Evaluator 16 3.00 3.10 2.55 3.10 3.00 Evaluator 17 3.00 2.20 2.60 2.15 2.45 Evaluator 18 3.00 2.60 2.55 2.40 2.50 Evaluator 19 3.00 3.80 3.30 2.80 2.70 Evaluator 20 3.00 2.60 3.00 2.40 2.15 3.00 2.78 2.73 2.55 2.63 3.00 2.34 2.55 2.38 2.34 3.00 3.38 3.00 3.06 2.903rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00Performance Score A lte rn at iv e Median 3rd Quartile 1st Quartile

G-10 TABLE G-9 Round 1 Scores - Cost Score Cost Score Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 3 2 4 3 4 Evaluator 2 3 2 3 2 2 Evaluator 3 3 2 3 3 2 Evaluator 4 3 2 4 4 4 Evaluator 5 3 4 3 3 3 Evaluator 6 3 1 2 5 4 Evaluator 7 3 4 5 5 5 Evaluator 8 3 2 3 5 3 Evaluator 9 3 2 4 4 4 Evaluator 10 3 2 4 4 4 Evaluator 11 3 2 4 4 3 Evaluator 12 3 4 4 5 4 Evaluator 13 3 3 4 4 3 Evaluator 14 3 1 2 3 1 Evaluator 15 3 2 4 5 3 Evaluator 16 3 2 2 4 5 Evaluator 17 3 2 3 4 4 Evaluator 18 3 4 2 4 5 Evaluator 19 3 3 3 4 3 Evaluator 20 3 2 5 4 5 3.00 2.00 3.50 4.00 4.00 3.00 2.00 3.00 3.75 3.00 3.00 3.00 4.00 4.25 4.003rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 1.001.502.002.503.003.504.004.505.00 Cost Score A lte rn at iv e Note that Cost Score scale is reversed. Median 3rd Quartile 1st Quartile

G-11 TABLE G-10 Round 1 Scores - Value Score Value Score Datum Current Preferred Design Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Softw are Alternative #4 Build on EC IMAGINE Evaluator 1 1.00 1.73 0.69 0.87 0.59 Evaluator 2 1.00 2.10 0.87 1.73 1.83 Evaluator 3 1.00 1.13 1.00 1.27 1.95 Evaluator 4 1.00 1.18 0.70 0.58 0.70 Evaluator 5 1.00 0.55 0.83 0.80 0.93 Evaluator 6 1.00 3.35 1.50 0.50 0.58 Evaluator 7 1.00 0.58 0.67 0.61 0.43 Evaluator 8 1.00 1.80 0.93 0.41 0.83 Evaluator 9 1.00 1.35 0.59 0.74 0.73 Evaluator 10 1.00 1.43 0.71 0.69 0.71 Evaluator 11 1.00 1.33 0.63 0.63 0.97 Evaluator 12 1.00 0.73 0.68 0.38 0.44 Evaluator 13 1.00 0.65 0.59 0.56 0.58 Evaluator 14 1.00 4.65 1.80 1.03 2.55 Evaluator 15 1.00 1.68 0.65 0.64 1.02 Evaluator 16 1.00 1.55 1.28 0.78 0.60 Evaluator 17 1.00 1.10 0.87 0.54 0.61 Evaluator 18 1.00 0.65 1.28 0.60 0.50 Evaluator 19 1.00 1.27 1.10 0.70 0.90 Evaluator 20 1.00 1.30 0.60 0.60 0.43 1.00 1.31 0.77 0.63 0.71 1.00 1.01 0.67 0.57 0.58 1.00 1.69 1.03 0.78 0.943rd Quartile Design Concepts Evaluator Median 1st Quartile D at um 0 1 2 3 4 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Value Score A lte rn at iv e Median 3rd Quartile 1st Quartile

G-12 Criteria Weight Rating Agency Acceptance 0.30 Technical Feasibility 0.20 Analytical Proficiency 0.20 Scalability 0.15 Responsiveness 0.10 International Credibility 0.05 Cost Implications 1.00 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Rating A lte rn at iv e 0 1 2 3 4 1.001.502.002.503.003.504.004.505.00 Rating A lte rn at iv e Note that Rating scale is reversed. Figure G-1. Round 1 evaluation performance and cost rating statistics.

G-13 Performance Score Cost Score Value Score Scores 0 1 2 3 4 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Performance Score Al te rn at iv e 0 1 2 3 4 1.001.502.002.503.003.504.004.505.00 Cost Score Al te rn at iv e Note that Cost Score scale is reversed. 0 1 2 3 4 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Value Score Al te rn at iv e Figure G-2. Round 1 score statistics. Figures F-3 through F-9 contain histograms showing the distribution of ratings (6 performance and 1 cost criterion), given by the 20 evaluators in Round 1, for the design alternatives. The shapes of the distributions give credence to the decision to use nonparametric rather than parametric statistics. That is because most of the rating distributions do not conform to a normal distribution, which is characterized by a symmetrical, bell-shape cluster of data around some mean. The presence of a normal distribution is a prerequisite for the use of parametric statistics. 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Agency Acceptance Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-3. Distribution of ratings for performance criterion – agency acceptance.

G-14 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Technical Feasibility Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-4. Distribution of ratings for performance criterion – technical feasibility. 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Analytical Proficiency Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-5. Distribution of ratings for performance criterion – analytical proficiency.

G-15 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Scalability Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-6. Distribution of ratings for performance criterion – scalability 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Responsiveness Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE FigureF-7. Distribution of ratings for performance criterion – responsiveness.

G-16 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating International Credibility Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-8. Distribution of ratings for performance criterion – international credibility. 0 5 10 15 1 2 3 4 5 R at in g Fr eq ue nc y Rating Cost Implications Alternative #1 Build on AEDT Alternative #2 Build on Existing Simulation Models Alternative #3 Federal Adoption of Commercial Sof tware Alternative #4 Build on EC IMAGINE Figure G-9. Distribution of ratings for cost criterion – cost implications.