Performance Measurement Tool Box and Reporting System for Research Programs and Projects (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

H- 1 APPENDIX H - PM Selection Wizard Questions, Answer Weights, and Comments Questions and Optional Response Choices 1. This agency would probably benefit most if in the near future the research program budget was: a. increased substantially b. increased slightly c. maintained d. reduced 2. Current sentiment among appropriation decision-makers is that near-future budgets for the research program will most likely be: a. increased substantially b. increased slightly c. maintained d. reduced 3. Elected officials and agency administrators who review performance measurement data are probably: a. highly focused on eliminating transportation impacts upon the environment b. very interested in all transportation-related environmental matters c. interested in transportation-related environmental matters d. believe that there are many other matters of considerably high importance to the agency 4. This agency’s research program currently uses annual customer surveys to measure satisfaction level and the degree that needs are being met: a. yes b. no, but we are already planning to initiate an annual survey c. no, but this is something that we would consider in the future d. no, this is not an option that we will consider right now 5. From the list below, indicate up to four statistics that the research program manager considers highly important for monitoring contractor/researcher participation and performance quality: a. % of research projects that result in implemented products b. % of research projects completed on or before originally proposed completion date c. % of research projects completed within the originally proposed budget d. % of research projects with all reports delivered within one year of project completion date e. total number of research contractors with an active project this fiscal year f. % of total research program contract dollars that was awarded to minority research contractors this fiscal year g. % of total research project funding supporting research performed within the agency (in-house research)

H- 2 6. From the list below, indicate up to four research product types considered to be of highest important to your agency at this time (there is overlap between some of the categories): a. environmental quality solutions or advancements b. congestion mitigating solutions or advancements c. safety solutions or advancements (general, including in work zones) d. operational cost savings for the agency e. traveler comfort advancements (ride quality, aesthetics, system security, sign legibility) f. quality of life advancements (traveler comfort items plus environmental protection and congestion mitigation) 7. From the list below, indicate the research product type of most importance to your agency at this time: a. basic knowledge advancement (with no immediate application to be implemented) b. technical advancements which improve quality of transportation system and/or agency operation c. agency policy and management advancements which improve quality of transportation system and/or agency operation 8. Agency resources available for tracking research program performance are best described as: a. not a factor in selecting the number and types of performance measures b. some resources will be diverted from other priorities, as needed, for this high priority activity c. limited resources are available d. it’s uncertain where the resources could be found 9. From the list below, indicate each item which is a current concern and a focus area for improvement in your research program at this time. a. Quality of research work b. Timeliness of implementation c. Timeliness of research completion d. Adequacy of research program funding 10. From the list below, indicate up to three agency performance indicators that are currently of the highest interest to the research program manager or agency administrator. a. Research office overhead expenses as a percentage of the total research budget. b. Total number of agency personnel involved in any manner with the agency research program. c. Percentage of research program customers reporting “satisfied” or “very satisfied” in a customer survey. d. Number of project needs statements submitted by agency personnel. e. Number of graduate students financially supported or otherwise involved in the research program.

Table of Weighting Scores Associated with Each Question Response Impact Points for Question Responses PM # 1 a 1 b 1 c 1 d ( 1 c o r 1 d ) & 2 a ( 1 c o r 1 d ) & 2 b ( 1 a o r 1 b ) & 2 c ( 1 a o r 1 b ) & 2 d 3 a 3 b 3 c 3 d 4 a 4 b 4 c 4 d 5 a 5 b 5 c 5 d 5 e 5 f 5 g 6 a 6 b 6 c 6 d 6 e 6 f 7 a 7 b 7 c 9 a 9 b 9 c 9 d 1 0 a 1 0 b 1 0 c 1 0 d 1 0 e 1 4 3 2 2 3 4 4 4 4 1 2 3 2 1 1 2 3 4 4 3 3 2 1 1 1 1 2 4 4 3 4 1 2 2 1 1 4 1 5 1 1 3 1 6 5 1 7 1 10 4 3 2 4 1 4 1 1 8 1 4 2 2 4 1 9 1 1 1 1 4 4 1 10 1 2 3 2 1 2 1 1 1 4 1 1 11 1 2 4 1 1 1

Table of Weighting Scores Associated with Each Question Response (cont.) 12 1 2 4 1 1 3 1 13 5 1 2 2 14 10 15 2 1 1 2 4 3 1 16 2 1 1 2 4 1 1 17 2 1 1 2 3 2 1 2 1 1 1 4 1 18 10 2 19 1 0 20 10 21 2 1 1 4 4 4 4 1 22 4 1 1 10 23 2 1 1 4 1 24 10 4 1 1

Table of Weighting Scores Associated with Each Question Response (cont.) 25 10 1 1 4 26 10 1 1 1 27 10 1 4 1 28 1 10 29 10 30 10 10 3 2 2 2 1 10 Note: Question 8 responses do not impact selection of individual PMs. These responses assist in analysis of selected sets of PMs at the conclusion of the wizard.

H- 6 PM Selection Wizard Comments and Triggers Trigger Displayed Comment Question 4 – Response d. selected, and Customer Satisfaction Level is manually selected as a performance measure. You indicated in a question response that your agency will not consider initiating customer surveys at this time. Your selection of the Customer Satisfaction Level performance measure may require that you initiate a customer survey. Question 4 – Response c. selected, and Customer Satisfaction Level is manually selected as a performance measure You indicated in a question response that customer surveys are not something that your agency is considering at this time. Your selection of the Customer Satisfaction performance measure may require that you initiate a customer survey. Question 8 – Response c. selected, and PM selections include #1, #2, #3 and #21 and more than 12 total PMs were selected. You indicated that resources are limited for tracking research program performance. The group of performance measures that you have selected may prove difficult to track within available resources. Question 8 – Response d. selected, and PMs selections include #1, #2, #3 and #21. You indicated that resources are very limited for tracking research program performance. The group of performance measures that you have selected may prove difficult to track within available resources. #21 is selected. Accuracy of the annual program benefit-cost ratio is improved by increasing the number of projects for which cost-saving benefits have been estimated. #1 is not selected. You have not selected Agency Costs Saved, which is one of the key performance measurements to be aggregated at the national level. You may wish to consider determining and entering this information into RPM Web even if it is not a formally used performance measures within your agency. #2 is not selected. You have not selected Lives Saved, which is one of the key performance measurements to be aggregated at the national level. You may wish to consider determining and entering this information into RPM Web even if it is not a formally used performance measures within your agency. #3 is not selected. You have not selected Reduction in Crashes, which is one of the key performance measurements to be aggregated at the national level. You may wish to consider determining and entering this information into RPM Web even if it is not a formally used performance measures within your agency. Performance measures are selected in only three or less of the five PM categories. You have selected performance measures in less than four of the five categories. A broader assessment of research program performance is obtainable by selecting at least one performance measure in additional categories. At least one performance measure is selected in each of the five PM categories. You have selected at least one performance measure in each of the five categories, which tends to assure a broader assessment of program performance.