Performance Measurement Tool Box and Reporting System for Research Programs and Projects (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

13 CHAPTER 3 – RESEARCH PERFORMANCE MEASURES The research performance measures gathered from the nationwide surveys were considered in conjunction with performance measure information available in the literature. A comprehensive list of research-related performance measures was compiled from the gathered information, and the perceived value of each metric was considered. The comprehensive list is shown in Appendix F. After analysis, the research team recommended 18 performance measures to the NCHRP panel for inclusion in the system to be developed. At the request of the panel, the research team agreed to expand the number of standard performance measures to be provided to 30. In addition, definitions for an additional 10 performance measures will be included to inform users of other metric possibilities. The panel believed that the larger group of performance measures was needed for the system to adequately address the broad range of needs and desires existing among the states. Finally, it was also decided that users should have the ability to manually incorporate any of these other performance measures, or their own agency’s unique performance measures, into performance measure reports available from the system. This utility has been provided in the RPM System. The selected standard performance measures and their definitions are shown in Table 3.

Table 3. Standard Research Performance Measures Included in the RPM System PM Number Short Performance Measure Name Definition Comment Outcome Measurements 1 Dollars Saved Estimated present value dollar savings in the cost of contract work, cost of agency-purchased materials, and cost of employee labor made possible by research products A core justification for research budgets. Very important to agency administrators and all funding appropriators. 2 Lives Saved Projected number of lives to be saved based on the number of fatalities associated with the problem prior to the product implementation and the estimated or determined effectiveness of the research products A core justification for research budgets. Very important to both agency personnel and all elected officials. 3 Crashes Avoided Estimated reduction in number of crashes based on the number of crashes associated with the problem prior to the research product's implementation and the estimated or determined effectiveness of the product A core justification for research budgets. Very important to both agency personnel and all elected officials. Output Measurements 4 Technical Products Number of types of research products improving design processes, specifications, or technical standards or practices Each product will either be a technical product, a management product, or a knowledge product. This is a general measure of the impact of the research program on the agency. 5 Management Products Number of types of research products improving the agency's management procedures, policies, and non-technical training Each product will either be a technical product, a management product, or a knowledge product. This is a general measure of the impact of the research program on the agency. 6 Knowledge Products Number of types of research products improving basic knowledge or understanding in the subject area without creating a specific technical or management product These are the products of basic research projects. This measure may be used to establish or maintain the desired level of basic research being funded by the agency. 7 Environmental Products Number of types of research products improving or protecting the natural environment Very important, and can be of primary importance to some state and federal appropriators and others.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 8 Congestion Mitigating Products Number of types of research products reducing or eliminating traffic congestion and other transportation system delays Very important to the general public and all elected officials. 9 Traveler Comfort Products Number of types of research products improving the physical or psychological comfort of the traveler or enhancing the aesthetic quality of the system or improving system security (safety products not included unless traveler comfort or well-being is improved in non-crash situations) Believed to be one of the most important factors to the traveling public. 10 Quality of Life Products Number of types of research products improving quality of life, which is defined as the total of those product types meeting the criteria for Environmental Products, Congestion Mitigating Products, or Traveler Comfort Products Important to the traveling public, the most important transportation agency customer. 11 Safety Products Number of types of research products improving design methodologies, traffic management, roadside safety devices, and any other innovation or enhancement for the transportation system which improves safety for anyone on or near the transportation system Safety is always a top priority. This is an indirect measure of the number of lives saved and reduced crashes made possible by the research program. 12 Cost-Saving Products Number of types of research products reducing the cost of contract work, cost of agency-purchased materials, and cost of employee labor This is an indirect measure of the amount of cost savings being obtained for the agency by the research project or program. 13 Research Reports Number of published research reports and other technical publications emanating from completed research projects during the evaluation year This measure combines two measures currently used by agencies: “Number of Papers Written as a Result of Program” and “Number of Research Reports Completed per Year.” 14 Graduate Students Total number of graduate students financially supported or otherwise involved in transportation research The value of the training given to future transportation professionals has been generally understated in the past. Resource Allocation Measurements 15 Dollar-Saving Projects Number of research projects pursuing lowered cost to provide the transportation system This measure monitors funding balance in the research program and the extent to which agency cost savings are being pursued.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 16 Safety Projects Number of research projects pursuing safety enhancements This measure monitors funding balance in the research program and the extent to which improved transportation safety is being pursued. 17 Quality of Life Projects Number of research projects pursuing improved quality of life This measure will be obtained by adding the number of projects including environmental products, traveler comfort products, and traffic congestion mitigating products. 18 Total Contractors Number of unique entities with research projects that were active for any length of time during the evaluation period If proposals are competitively awarded, this is an indirect measure of competitiveness. 19 Minority Contractors Percentage of total research program contract budget that is awarded to minority universities, as defined by the US Department of Education and applicable federal regulations A federal requirement, reported at least annually. 20 In-House Percentage Percentage of the total funding for research projects being performed by agency personnel This can be an indicator of growing or declining in-house technical strength. Efficiency Measurements 21 Benefit-Cost Ratio Total present value dollar savings associated with the project(s) compared to either the total present value cost of the project(s) plus implementation effort(s) or to the total present value cost of the fiscal year research program plus related implementation efforts. The system report generator selects the cost basis and enters cost data. A key efficiency measurement for state and federal budget appropriators. 22 % Administrative Costs Dollar value of program overhead expenses divided by the total program cost An internal efficiency measurement. 23 % Requests Funded Number of projects funded divided by number of projects requested A lowering trend indicates probable need for additional research funding. 24 % Projects Implemented Number of projects with at least one product implemented (completely or partially implemented) divided by total number of projects completed during the evaluation period An indicator of quality in the project selection process and research project execution.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 25 % Projects On Time Number of projects completed on/before the scheduled completion date divided by total number of projects to have been completed during the evaluation period This target should probably be around 80 percent due to the nature of research. A lower percentage can indicate generally poor contractor efforts in creating proposal work schedules. 26 % Projects within Budget Number of projects completed within budget divided by total number of projects completed during the evaluation period This target should probably be around 80 percent due to the nature of research. A lower percentage can indicate generally poor contractor efforts in creating proposal budget estimates. 27 % Project with Reports Number of projects completed during the evaluation period (FY one year prior) for which all research reports have been submitted within one year of project completion divided by the total number of projects completed during the evaluation period This is a challenging area for most research programs. Monitoring performance and having a target can be used as a tool for the research manager to encourage or require improved contractor performance. Stakeholder Measurements 28 Customer Satisfaction Number of customers reporting satisfied or very satisfied on survey divided by total number of customers surveyed Variations of surveys were reported on survey responses from several states. It is believed that all stated needs can be addressed by the definition of this PM. 29 Agency Participation Number of agency personnel involved in the program overseeing projects, serving on committees, assisting in project selection, etc. Most research programs need the participation of large numbers of agency personnel from outside of the research office. There are a number of benefits to the agency derived from this participation. This number should be provided to agency administrators. 30 Project Needs Statements Number of project needs statements submitted by internal customers This is a key indicator to research program managers for several reasons, particularly in that it shows the degree to which agency personnel understand that research provides solutions to everyday problems.

18 One of the considerations of the research team was to attempt to select performance measures which would not only meet the needs of state transportation agency administrators and research program managers, but that would also provide for the needs of secondary customers such as contract researchers, stakeholders such as state and federal legislators, and other state-level research professionals. It was not cost effective or practical to include every possible performance measure in the system being developed, but the research team endeavored to provide a set of performance measures which was as comprehensive as possible. The standard performance measures are divided into five different types in Table 3. The five types of measures are outcome, output, resource allocation, efficiency, and stakeholder. Outcome measures assess the extent to which a research project or a product of a research project achieves a desired result such as cost savings or reducing crashes. Output measures count the number of deliverable units related to a specific attribute, examples being the number of research projects which improve safety and the number of products from projects which positively impact the environment. The third type of measure deals with resource allocation. Resource allocation performance measures primarily capture the deployment of agency dollars, such as the percent of research funding awarded to minority contactors or the number of research projects being funded in attempts to improve transportation safety. In contrast, efficiency measures, the fourth category, are rates or ratios which compare what is accomplished to the effort expended. Examples of this type of measure are the percent of research products being implemented by the agency and the percent of research projects being completed within budget. The final type of performance metric is the stakeholder measure. Stakeholder measures gage the involvement of customers in the research process as well as their level of satisfaction. These performance measures include the percent of satisfied customers, number of participating agency personnel, and the number of project needs statements submitted. By subdividing the performance measures into these five categories, the user is given the opportunity to better balance the selected set of performance measures to be used.