National Academies Press: OpenBook

Performance Measurement Tool Box and Reporting System for Research Programs and Projects (2008)

Chapter: Appendix G Performance Measurement 101 Narration

« Previous: Appendix F Comprehensive List of Performance Measures
Page 125
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 125
Page 126
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 126
Page 127
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 127
Page 128
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 128
Page 129
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 129
Page 130
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 130
Page 131
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 131
Page 132
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 132
Page 133
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 133
Page 134
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 134
Page 135
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 135
Page 136
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 136
Page 137
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 137
Page 138
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 138
Page 139
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 139
Page 140
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 140
Page 141
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 141
Page 142
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 142
Page 143
Suggested Citation:"Appendix G Performance Measurement 101 Narration." National Academies of Sciences, Engineering, and Medicine. 2008. Performance Measurement Tool Box and Reporting System for Research Programs and Projects. Washington, DC: The National Academies Press. doi: 10.17226/23093.
×
Page 143

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

G- 1 APPENDIX G – Performance Measurement 101 Narration Welcome to the Performance Measurement 101 tutorial, PM 101. It is important for the user to understand the basics of performance measurement when using the RPM System. Many users may also benefit from an orientation to the commonly used research performance measures (PMs) and from discussions about the somewhat unique aspects of research performance measurement. To assist first-time users, PM 101 includes an overview of the entire RPM System and information about each of the system tools. Click on Next to proceed through PM 101 from the beginning or on the information of particular interest in the drop-down box below. RPM System Overview The RPM System consists of two major components: RPM-Tools and RPM-Web. Each includes a number of resources designed to assist you in calculating the return on research investment. The following chart provides a complete list of the resources available and indicates where to find each one. The RPM System is composed of a website called RPM-Web located at the website indicated below, and RPM-Tools, a CD-ROM designed to complement the website while operating independently. The RPM System allows assessment of performance on an individual research project basis or on an annual research program basis. The RPM System also provides the user with a variety of options for reporting the performance being measured. RPM-Tools is a collection of resources and automated tools to assist the user in research performance management. These items are linked at appropriate locations to facilitate navigation. The major elements of RPM-Tools are: Home – Returns you directly to the opening screen of this CD-ROM. PM 101 – The PM 101 tutorial is an introduction to performance measurement principles, application of these principles to research program activities and products, and general instructions in use of other RPM-Tools. PM Wizard – The wizard is an analysis tool that offers guidance during the process of selecting research Performance Measures for an agency. Data Entry – Data entry screens for entering product-level specifics. Only product-level reports can be generated from the RPM-Tools CD.

G- 2 Benefit Calculation – A catalog of benefit estimation examples from which users can select an automated worksheet for creating their own estimations. PM Reports – Generate a report summarizing all information entered and created concerning a single research product. The report can be saved for later access or uploaded to RPM-Web. Resource Links – A listing of sources for statistics and other information frequently needed during benefit estimation. Glossary – Definitions of important terms used within the RPM System. RPM-Tools assists the user in assessing performance of a research program, project and/or product. The key tool in this set is the PM Wizard, a decision tree that guides the user through the initial selection of performance measures. There are also user-friendly calculation tools for appropriate performance measures. The user may navigate the decision tree and complete the performance measures which are designated as appropriate for the product. Information entered or calculated using RPM-Tools cannot be saved to the CD-ROM, but it can be saved to the user’s hard drive for later viewing or completion of the work session. RPM-Web is the dynamic hub of the RPM System. While offering virtually all of the features of RPM-Tools, RPM-Web is the place where agencies store research performance measurement information. RPM-Web also offers a suite of pre-programmed reports that summarize research performance information for individual research projects, categorized groups of research projects, an agency’s entire annual research program, or the nationwide annual research program. Access to the RPM-Web is restricted. Research managers will designate key individuals to enter the necessary agency and research program information. This process is greatly facilitated by periodic downloading of project information from the Research-In-Progress (RIP) database maintained by the Transportation Research Board. RPM-Web allows both direct user input of information and PM calculation, and it also accepts uploaded product-level information from RPM-Tools. Once an annual research program is established on RPM-Web, authorized individuals may input individual project and product information. Most performance measurement information is introduced into the RPM System at the research project and research product levels. An agency may choose to include and evaluate all of the agency’s research projects or only a sub-set of their projects. Unlike RPM-Tools, multiple users can access project and product information on RPM-Web. A project or product can remain open on RPM-Web until such time as all of the information is entered. Performance measure determinations which are completed are integrated into the program-level assessment for each state agency. RPM-Tools and RPM-Web are designed to allow flexibility in how they may be utilized. Agencies may choose to use both RPM-Web and RPM-Tools internally for calculating performance measurement information. They may alternatively decide to provide RPM-Tools

G- 3 and/or information-entry access to RPM-Web to their research contractors. Requiring research contractors to estimate potential benefits at the end of their research projects has potential to greatly assist the sponsoring agency in performance measurement. Necessary project information includes project title, number, total budget, and beginning and end dates. The user also identifies anticipated products to result from each research project. It is at the product level where benefits of implemented research are estimated and a number of performance measurement assessments are performed. Access to RPM-Web is assigned by the agency research program manager. A log-in and password security system is used. Several levels of access are available for assignment by the research program manager. The RPM System offers research program managers and agency administrators a powerful means of establishing and maintaining a performance measurement strategy for the agency’s research program. 1.0 What is performance measurement? Defining performance measurement and management Performance measurement may be defined as the act of comparing results to specified standards. These standards may be established internally or imposed by an external entity. Specific performance measures are indicators or metrics that are used to gauge, quantitatively and/or qualitatively, the impact of activities and actions. Collecting quantitative data is the preferred method of measurement. However, uses of qualitative information and hybrid measures have become increasingly popular. Performance management is the use of techniques and processes to set goals, identify appropriate performance measures, assess the impact of initiatives and effectively communicate the information internally and externally. Performance management assists both the operational and strategic planning structure of organizations. Specific performance measures are indicators, or metrics, used to gauge the impact of activities and actions. These indicators can be quantitative or qualitative, or both. Why is measuring performance beneficial? Performance measurement is no longer reserved for organizations seeking to produce a profit or satisfy stockholders. Interestingly, performance measurement has seen its most recent growth in applications such as government programs, non-profit organizations, health care and customer service areas of business. Although quantitative data is still the preferred method of measurement, qualitative information as well as hybrid measures which integrate the two methodologies have become increasing popular. Performance measurement serves many purposes and can speak to a number of audiences. Performance measurement can be any of the following:

G- 4 ƒ Program justification ƒ Communication tool ƒ Management metric ƒ Motivational technique ƒ Feedback mechanism ƒ Process indicator In the past, performance measurement has sometimes been saddled with a negative reputation. Activities and output were measured for the sake of measurement and often failed to identify the impact or quality level of a particular operation or process. Performance measures and their associated results were used as negative motivators with employees. Soon the sentiment became that certain jobs were too complex to be measured or related to development activities which were not expected to have performance goals or measurements imposed on the work. The thought was that some things should be exempt from performance measurement. Even though that argument still exists, it has not stopped progress in the area of performance measurement. With the level of information which is available to stakeholders as well as the general public through mainstream media and the internet, it is critical that we feel comfortable with identifying appropriate performance measures and interpreting the results. Transportation research is not exempt from the application of performance measures. Stakeholders, such as federal and state agencies which support transportation research, use performance measurement to determine funding allocation as well as a resource for appropriation requests from Congress and state legislators. 2.0 What are the characteristics of a good performance measure? And what are the basic types? The value of a performance measure to the organization is determined by the extent to which it possesses the following characteristics. Relevance - The core mission of the organization is impacted by the factor being measured. Measurability – An objective means of determining tangible degree of success is available. The method used to determine performance or to estimate future impact is straight forward. Clarity - The meaning and application of the factor being measured are easily understood by both those striving to achieve the standard and by those evaluating performance. Highly technical terminology should be avoided. Feasibility – It is practical to obtain the necessary information within the resource limitations of the organization.

G- 5 Credibility – The information being collected and the means of collecting it are plausible to users of the performance information. In order to be credible, performance determinations dependent upon the use of estimations should be based on conservative estimates. These estimations should be provided by the most knowledge personnel available. The source of the estimation must be documented. Budget Importance – The information being gathered provides useful information to budget decision-makers. Proper resource allocation is a primary management tool for optimizing organizational success. Impact on Collaboration – The performance measurement provides the means for focusing different organizational entities on a common goal. Major organization objectives usually require cooperative effort between organizational units or programs. The above characteristics are all desirable for transportation research performance measures. The core mission of a transportation agency could be said to be to provide safe and efficient movement of people and goods. Therefore, transportation research performance measurement ideally will include measurement of factors associated with transportation safety and cost-effectiveness in agency operations. There are different types of performance measures. Performance measures can be placed into one of five categories. Each performance measure included in the RPM System has been assigned to one of these categories. Outcome Measures capture the extent of desired results provided by the activity or area being evaluated. Transportation examples of outcome measures are number of lives saved, reduction in number of crashes, and financial savings. Output Measures count the produced or delivered units derived from an operation being evaluated. Examples of research program output measures include number of technical products being developed and number of environmental products being implemented. Resource Allocation Measures quantify the deployment of personnel and financial assets. This category of performance measure is usually considered at the program and project levels. Examples of resource allocation measures include the number of research projects addressing specific areas, such as safety or quality of life, and also measures which track contractor demographics. Efficiency Measures are rates or ratios which compare what is accomplished to the amount of effort or opportunity involved. Examples of efficiency measures are benefit-cost ratio, percent of budget allocated to administrative costs, percent of projects completed within budget, and percent of research products being implemented.

G- 6 Stakeholder Measures determine the extent that customers are involved and their level of satisfaction. The goal of these measures is to capture the impact of the research program on the intended customer as well as to gage the internal involvement of the agency. Examples of stakeholder measures include percentage of satisfied customers and the number of agency personnel participating in the research program. Performance measures are often grouped by category in performance measurement reports. It is also helpful to consider performance measures by category during the process of selecting measures for an agency. A set of performance measures which involves only a couple of performance measure categories is unlikely to provide a comprehensive view of performance. 3.0 How does research performance measurement benefit the research program and help the program manager? Measuring performance can be a powerful tool for the transportation research program manager. Research performance measurement provides four primary benefits for both the research program and the program manager. Improved Communication – Performance measurement provides a communication tool for articulating research program objectives and the results being obtained. It is not uncommon for transportation research program managers to have limited opportunities to interface with agency administrators. Also, the turnover rate of top agency administrators increases difficulty in keeping the research program aligned with administration objectives. The process of establishing performance measures often allows direct interaction with top administrators, thereby assuring that desires of administrators are heard and understood. The research program manager is also assured that accomplishments of the research program are communicated appropriately. Program Justification – Performance measurement documents the value of the research program to the organization. Transportation research programs receive funding primarily through state legislative action. As state funding levels are usually limited, organizations closely evaluate their allocation of funding between various programs. Documentation of the financial, safety and environmental benefits made possible by the research program can be a critical factor in determining continued funding levels. By utilizing a performance measurement system which addresses outcome, output, resource allocation, efficiency and stakeholder metrics, a research program can demonstrate the effective use of funding on multiple levels. By

G- 7 using the different types of measures, the research program can speak to concerns of a diverse set of stakeholders. Improved Performance – Performance measurement cultivates an environment of accountability where high performance standards are common and achievement of high performance is rewarded. Performance management has always been an integral part of business operations in the private sector due to the required reporting to stockholders and other stakeholders. Specific industries have developed elaborate systems to assist in managing quality and improving performance. Total Quality Management (TQM), Balanced Scorecard and Six-Sigma are all examples of programs which address performance management and which have successfully improved performance within organizations. Feedback – Performance measurement provides the research program manager important statistics which allow timely revision of the program. In addition to tabulating outcomes, such as reduced fatalities and operational financial savings, there are a number of other important statistics for a research program manager to monitor if a healthy research program is to be maintained. Customer satisfaction level is a good example. Customer perceptions will affect the future of the program whether or not the perceptions are correct. Trends in various performance measure statistics can indicate changes occurring within the program that need to be addressed. For example, should the percentage of projects completed within budget be in a clearly downward trend, the research program manager would be wise to investigate and determine the cause for the trend. Are agency project managers beginning to require work beyond the original work plan? Or perhaps is there evidence that contractor budgeting is weak, thereby causing the problem? Or have some contractors begun showing lower budget estimates to increase the chances of getting the work, but planning to obtain additional funding later? Each of these possible scenarios would require different actions from the program manager to remedy. But the problem developing within the program would likely have gone unnoticed for much longer if not for performance measurement statistics. 4.0 What research performance measures are commonly used? Definitions and basic information about forty different research performance measures are provided here. Thirty of the performance measures are programmed for automated user assistance should they be selected by the user. The remaining performance measures must be manually entered into appropriate performance reports if they are to be used and reported. The thirty research performance measures include all of those reported to be commonly used by state transportation agencies in 2004. As discussed in Section 2.0, these

G- 8 performance measures are each assigned to one of five categories. The short names for all performance measures are listed by category. Click on the short name to obtain the full name, definition and additional information about the performance measure. You may also browse information on all included performance measures by clicking View All PM Tables. 5.0 What performance measurement tools are provided in the RPM System? What will they do for me? RPM System includes a variety of tools to assist research program managers in establishing and conducting performance measurement within their programs. While most tools within the system are available in both RPM-Tools and RPM-Web, the several exceptions are indicated in the brief tool descriptions provided below. PM 101 (available in RPM-Tools only) – This tutorial is an introduction to performance measurement principles and the application of these principles to research program activities and products. It also provides general instructions in the use of tools available in the RPM System. PM Selection Wizard (available in RPM-Tools only) – The wizard is an analysis tool that offers guidance during the process of selecting research PMs for an agency. Benefit Estimation Catalog – This catalog provides a series of benefit estimation examples from which users can select an automated worksheet for creating their own benefit estimation. Resource Collection – The resource collection is a listing of sources for statistics and other information frequently needed during the process of estimating benefits to be derived from research products. The listing is composed primarily of resources that may be found on websites, and URLs are provided for quick access. Present Worth Calculator – This tool converts future or historical monetary amounts into present-day dollars. Glossary – The glossary defines important terms used within the RPM System. Product Report – A report may be generated which summarizes all information entered and created concerning a single research product. This report may be saved outside of RPM-Tools for later access. The information may also be uploaded to RPM-Web. Project and Program Reports (available in RPM-Web only) – This suite of pre- programmed reports allows authorized users to obtain summaries of all information entered or created in RPM-Web concerning a single research project, an entire agency research program, a sub-set of projects from a single agency’s research program, and all projects nationwide. This reporting function may be accessed by clicking on the Reports tab in the main navigation of RPM-Web.

G- 9 6.0 How do I get started? A. Selecting performance measures A universally ideal set of research performance measures does not exist. The most appropriate set of research performance measures for a given agency may well differ from those of all other state research programs. Each agency has somewhat different goals and visions, and every research program has somewhat different strengths and opportunities. So, there will naturally be uniqueness among the agencies in research performance measurement. However, there are several performance measures that warrant strong consideration by every research program manager. These include the three outcome performance measures, Lives Saved, Crashes Avoided, and Dollars Saved, all of which are tied closely to the collective mission of AASHTO transportation agencies. Several factors should be kept in mind during the process of selecting research performance measures for an agency’s research program. ƒ Alignment with agency strategic objectives – The set of selected performance measures should include measures which monitor the success of the research program in directly supporting applicable agency goals. ƒ Value as a management aide – Every research program manager depends upon certain “vital statistics” for generally monitoring research program health. These may differ from manager to manager, and they are often monitored informally. These are good candidates for performance measures if it is desired to bring emphasis to them within the organization and to insure their periodic review. ƒ Availability of resources to track performance – Measurement of some performance measures is a time-consuming endeavor. Most programs must prioritize and select only those performance measures of most importance to the agency and the research program manager. ƒ Availability of the information necessary to track performance – There may be performance measures that are used in other states, and that would be great for your agency, except that the data to support it is not readily available in your agency’s organization. Look closely at the minimum data inputs required for a performance measure before selecting it for your program. ƒ Scope of performance measurement – Before completing performance measure selection, an important question to ask is if the set of selected performance measures is broad enough in coverage to adequately assess overall program performance. In addition to outcome measures, including one or more efficiency measures, output measures, stakeholder measures, and/or resource allocation measures can assure a much more comprehensive annual analysis of the program.

G- 10 ƒ Agency approval process – Every agency has an established procedure for establishing formal performance measures. Insuring familiarity with this process, particularly regarding the frequency with which revisions might be possible, can help avoid time-consuming errors and frustration in later years. ƒ PM Selection Wizard – The wizard is a tool which can quickly provide the user some suggestions during the process of selecting research performance measures. In addition, the wizard provides the user with information about the relative usefulness of each performance measure in the pre-programmed standard report formats. B. Selecting performance goals Establishing appropriate performance goals or targets is just as important as selecting a good set of performance measures. The goals should be challenging yet achievable. Determining levels of performance which will be a challenge yet achievable normally takes some homework and serious consideration. The first step should be to assess the research program’s performance over the past several years in each new performance area being considered. This assessment should be done by those individuals who will make these measurements during the actual measurement process. It is important to discover during this process the degree of difficulty involved with obtaining the necessary information and making the actual measurements. The results during previous years, together with the program manager’s assessment of the degree of effort expended to obtain the levels of performance obtained in previous years, will establish a performance baseline for each new performance measure. The next step is for the research program manager to establish the ultimate performance goals for the program to achieve in new performance measurement areas. The program manager should establish these goals cooperatively with the managers or individuals who will be primarily responsible for goal achievement, whenever this is feasible. The program manager may also wish to consider generally accepted goals, or goals used by other similar agencies. However, this can lead to serious errors in goal selection if basic differences exist between the manager’s program and the programs in other agencies. If current performance is not meeting some of the ultimate goals that were determined in step two, the final step is to assess how long it may practically take to achieve the desired level of performance in each of these areas. It may be that the ultimate goals will require more than one year to achieve. Again, input from those directly responsible should be obtained and considered. Caution should be taken considering the input from those who will be responsible for goal achievement. Highly motivated employees are just as likely to overestimate what achievement is practical as some may be to underestimate it. Following this three-step process will result in good initial selections for most new performance measure goals.

G- 11 C. Entering information into the RPM System There are several ways provided to enter information into the RPM System database. All required information may be directly entered using RPM-Web add/update screens, provided that the user has authorized access to these screens. Additionally, any user of RPM-Tools may enter all information pertinent to a given product into screens provided in RPM-Tools. This information can then be saved outside of RPM-Tools, or the information may be uploaded to RPM-Web if the user has an appropriate RPM-Web access role. Basic annual research program information must be provided to RPM-Web to establish the annual research program in the system database. This must occur prior to entry of project information for that program. Project information, including identification of anticipated products, must be established prior to entering detailed product information. Authorized access to RPM-Web is required to enter any information into the system’s database. Five of the nine access roles allow entry of certain types of information. The research program manager assigns access roles. All access roles and levels, and their access to RPM-Web add/update screens, are shown below. As can be seen, each research program manager may elect to grant principal investigators and/or contractor administrative offices level two access, which allows entry of certain information for their research projects. However, information entered through these access roles is placed in a pending status until approval by one of two agency roles incorporates it into the RPM-Web database. User entry of specific research project information is being greatly facilitated by the Transportation Research Board, which is providing periodic downloading of project information from the Research-In-Progress (RIP) database. For this reason, the agency will benefit considerably from routine and early entry of information on new projects into RIP. When project information from RIP has been downloaded, the authorized user establishes the downloaded information into the RPM-Web database by simply reviewing and indicating that the information is accurate. This process allows the user to correct potential errors, particularly involving information which matches RPM-Tools uploads to the appropriate RPM-Web project file. An agency may choose to include and evaluate all of the agency’s research projects or only a sub-set of their projects. Unlike RPM-Tools, multiple users can access project and product information on RPM- Web. A project or product can remain open on RPM-Web until such time as all of the information is entered. Performance measure determinations which are completed are integrated into the program-level assessment for each state agency. When and how is the data entered into the RPM system? On an annual basis, the state agency which is the designated transportation research entity will initialize the RPM system for the fiscal year. The data in the system can remain the same from year to year with the exception of the budget, number of projects and fiscal

G- 12 period. Once this operation is completed, authorized users can begin loading project information into the system. The agency can enter project and product data at any time during the year. It is recommended that project information be loaded into the system for a fiscal year at the beginning of that year. Adding a project to a program is simple, only requiring the user to enter general project information and answer several questions relative to the anticipated impact of the project. The following is a list of the questions which are presented at the project entry level: ƒ Is a purpose of the project to improve safety, reducing crashes, injuries or fatalities? ƒ Is a purpose of the project to reduce the cost of providing the transportation system, thereby saving or stretching tax dollars? ƒ Is a purpose of the project to positively impact the environment through recycling, improving air quality, or by other means? ƒ Is a purpose of the project to improve traveler comfort by reducing traffic congestion, improving security, improving ride quality, or by other means? Once a project has been loaded, the user can identify the products which are associated with the project. The user will indicate whether the deliverable is a product or a report. Additionally, the user will designate the product as being a technical product, a management product, or simply basic knowledge advancement. After the project is completed, or whenever a product is delivered, the user will re-enter the system to close out the product. Based on the information initially entered at the product and project levels, the user will be prompted with only those questions which are pertinent to the specific product. Once the user closes out the product(s), the project level will need to be completed. The project and product data will be automatically integrated into the program level. The following sections provide some additional information relative to the particular levels of information in the RPM system. Entering Program Level Data The RPM System requires the state agency or a designee to enter information related to the transportation agency, the annual research program, individual projects and specific products. The program level information is set-up on an annual basis and serves as the foundation for all of the information entered relative to a particular fiscal year. Information contained at the program level includes, but is not limited to, the following: ƒ State agency name ƒ Contact information for the research program manager ƒ Overhead or indirect rate ƒ Number of projects ƒ Contractor or grantee information ƒ Total annual research budget

G- 13 ƒ Total number of project needs submitted for funding consideration ƒ State agency strategic objectives ƒ Fiscal year definition Entering Project Level Data The project level of the RPM System allows the user to enter the projects which have been designated to be included in the database. The agency decides if all of their funded projects will be included in the systems or only a subset of projects. Information contained at the project level includes, but is not limited to, the following information items. ƒ Project title ƒ Budget amount ƒ Project period ƒ Contractor or grantee ƒ Designation of completion ƒ Type of project Entering Product Level Data The product level of the RPM System allows the user to assign products to a specific project. These products are either considered actual products or reports, which are considered an administrative type of deliverable. The system is designed to integrate product information into the project level and subsequently into the program level. Most of the performance measurement information the user enters into the system is completed at this level. Of the three levels, this level is the most complex because the user may have to calculate benefits based on the implementation strategy as well as assess the specific impact in a number of areas such as traveler comfort, environmental, safety, etc. Information contained at the product level includes, but is not limited to, the following items. ƒ Name of product ƒ Type (report or product and technical, management, or basic knowledge) ƒ Implementation status of product ƒ Completion status of product (completed, waived or not complete) ƒ How product implementation affects safety and/or operational cost ƒ Calculation of estimated benefits D. Measuring performance The RPM System provides for tracking both quantitative and qualitative performance measures. Qualitative measures are converted to quantitative measurement through counting the number of qualitative impacts which are pursued and the number of qualitative products that are implemented.

G- 14 Most performance measurement information is entered at the project and product levels. Responses to questions at the time of project and product information entry provide the bases for a number of performance measurements. However, the outcome measures must be performed through an estimation of benefits to be obtained from each implemented product. Benefit estimates should be performed by the most knowledgeable individuals available. These estimates may be performed entirely within the agency sponsoring the research, or they may be performed by the contractor at the request of the sponsoring agency. Benefits should be estimated conservatively. And it is imperative that sources for statistical information and expert estimates be documented at the time that the estimate is made. Without source documentation, the estimate will not be credible to a questioning user of performance measure information. A catalog of example benefit estimations with automated worksheets is provided. The user may select an example to follow when creating their own benefit estimate. The user can insert the information relevant to their specific product into the example, and working formulas will determine their benefits. Alternatively, the user may choose to use a blank worksheet provided within the Benefit Estimation Catalog. Additionally, the user will find a Resource Collection composed of sources for statistics and other information commonly needed in estimating benefits. Simply go to the Resource Collection and browse the contents, or search by key word to make locating desired information much quicker. The contents of the Resource Collection are listed by category. The categories are infrastructure, safety, freight transportation, passenger travel, registered vehicles and vehicle-miles traveled, economy and finance, and energy and environment. A resource is listed in multiple categories when appropriate. Another aide provided to users is Discount Rate Guidance. Credible business investment analyses extending over a period of years require use of a discount rate to account for the time value of money. Estimations of monetary benefits being derived from research should also consider the time value of money. Research benefit estimates are the bases of major investment decisions within the agency, just as investment analyses are in private business. While discount rates in private business usually include a minimum acceptable profit percentage, as established by business owners, non-profit organizations often base discount rates on the interest rate they pay for long-term loans. This interest rate is called their cost of capital. Other agencies base their discount rate on other factors. It is recommended that each research program manager, in consultation with their agency’s financial officer, establish the discount rate to be used in their benefit estimations. The discount rate to be used should be reviewed annually.

G- 15 Additional information and recommendations concerning discount rate selection may be found at the website of the Office of Management and Budget (OMB) at the URL shown. A default discount rate of 2.5 percent is shown on the automated benefit estimation worksheets. It was selected based upon the 2005 real discount rates recommended by the OMB for cost-effectiveness analyses. Historical OMB recommendations for real discount rates may also be found by following the appropriate link provided at the website above. It is also required during benefit estimation for the estimator to select an Anticipated Life of Products before Obsolescence. As when other types of estimates are necessary, three recommendations are made to improve credibility. First, the most knowledgeable person available should make the estimate. Second, that person should make the estimate on the conservative side of the probabilities foreseen. And third, the identification of the person providing the estimate should be documented. To assist in this estimation, the following table provides suggested ranges from which the useful life of a variety of research product types may be selected. These ranges are provided as guidelines only, and a useful life outside of this range may be used when warranted. The suggested ranges were developed from responses to an opinion survey taken at the 2004 National AASHTO Research Advisory Committee Meeting. It is understood that the next generation of a technology is often based upon the technology of the product being replaced. Since the next generation technology would not have been possible without the development of the replaced product, the case can be made that some degree of benefits continue to be derived from a given innovation after the product itself has become obsolete. However, for purposes of determining benefits for research performance measurement, benefits should be considered to cease at the point that the specific product becomes obsolete. This principle is also in the spirit of conservatism, thereby increasing credibility. The Present Worth Calculator provided can help the user quickly discount future monetary amounts occurring in future years as well as adjust monetary transactions occurring in prior years. The user simply enters the monetary amounts involved, the year in which each occurs, the desired present worth year, and the discount rate to be used, and the calculator does the rest. E. Creating performance reports A series of pre-programmed reports are provided within the RPM System. Access to RPM-Web reports varies according to the access role of the user, as shown in the table below. There are no restrictions to access to reports available within RPM-Tools. The PM Selection Wizard Report and the Individual Product Report are the only reports available within RPM-Tools.

G- 16 All reports except for the PM Selection Wizard Report may be modified by the report creator. The user has the option to delete any performance measurement data from any performance report by deleting entire rows or entire columns of information from the pre- programmed format. Performance measures which were not selected by the agency will not appear on performance reports created for that agency. A modified report format on RPM-Web, created by deleting rows or columns of information, is automatically saved for later use by personnel in that agency. RPM-Web reports displaying performance information may be named and saved to the RPM-Web database by clicking on the Save Report button on the report page. These reports may then be accessed by anyone with authorized access. RPM-Tools reports may be saved on the user’s computer, outside of RPM-Tools. Reports generated in RPM-Web and RPM-Tools may be printed by clicking on the Print Report button on the report page. A description of each RPM System report follows. PM Selection Wizard Report A listing of performance measures selected for use by an agency may be saved or printed for user convenience. The list is sorted by performance measure category. Individual Product Report The default format of this report contains all product information available in the database for a selected research product. This report also displays documentation of benefit estimation sources and calculations. Individual Project Report The default format of this report contains all project and product information available in the database for a selected research project. The primary purpose for this customizable report is to provide the Research Program Manager one or more report formats suitable for either internal research office or to meet the information needs of individuals or organizations external to the research office. A feature of this report, like the Individual Product Report, is that it includes documentation of the method of benefit calculations. In this report, information on each product entered into RPM-Web is displayed. The sponsoring agency user may also add one or more unique performance measures, goals, and results achieved in free form text fields provided for unique performance measures. The title of the report may also be customized. The Research Program Manager has the option to save this customized report configuration on RPM-Web to make it available to other system users within his or her

G- 17 own agency. Upon opening a saved report configuration, an authorized user obtains up to the minute information pertinent to that report unless the report was saved as a static report. The Research Program Manager may also save a report to their computer for attaching to emails, preserving data reflecting that report date, or otherwise handling. Users from another agency will only be able to access the default format of the Individual Project Report. The default format of this report is the report available from RPM-Tools. Multiple Project Report The default format of this report contains most project and product information available in the database for each research project sponsored by the requestor’s agency. The primary purpose for this customizable report is to provide the Research Program Manager one or more report formats suitable either for internal research office use or for providing information to individuals or organizations external to the research office. This report may also be customized to include only projects associated with one or more agency strategic objectives, agency-defined categories, funding type, or national standard categories from the fiscal year being reported. The title of this report may be customized. As with the Individual Project Report, the Research Program Manager may save a configuration of this report on RPM-Web to make it available to other system users within his or her own agency. The user may also save a specific report to his or her computer. State Research Program Performance Report The default format of this report contains two tables of information describing the performance of the agency’s entire research program for the requested fiscal year. The first table includes program efficiency information; and the second table includes program effectiveness information. The primary purpose for this customizable report is to provide the Research Program Manager one or more report formats for use in monitoring program performance or for reporting program performance to agency administrators. The report for a given agency is available only to system users belonging to that agency. The prior fiscal year’s performance is displayed along with the requested fiscal year’s targeted and actual performance. A user may customize the default report format by eliminating rows or columns of information. The sponsoring agency user may add one or more unique performance measures and results in free-form text fields. The title of the report may also be customized. The Research Program Manager has the option to save this customized report configuration on RPM-Web, to make it available to other system users within his or her own agency, or he or she may save the specific report to their computer.

G- 18 State Research Program Impact Report The default format of this report provides information about an agency’s research program that is pertinent to three common and primary research program objectives: to save lives, to reduce costs of providing transportation, and to improve quality of life. The default report includes an indication of the amount of research being performed toward each objective, the outputs and outcomes which resulted, and indicators of the efficiency of the agency’s entire research program. This primary purpose for this report is to succinctly provide information needed by decision-makers responsible for requesting or approving an agency’s research program funding level. The report for a given agency is available only to system users belonging to that agency. A user may customize the default report format by eliminating rows or columns of information. The title of the report may also be customized. The Research Program Manager may save this customized report configuration on RPM-Web, to make it available to other system users within his or her own agency, or he or she may save the specific report to their computer. National Research Program Impact Report The default format of this report provides information about the effect of the combined state research programs. Like the similar agency-level report, this report focuses on three common and primary objectives of every research program: to save lives, to reduce the costs of providing transportation, and to improve quality of life. The default report includes an indication of the amount of research being performed nationally toward each objective, the outputs and outcomes which resulted, and indicators of the efficiency of the nationwide research program. The primary purpose for this report is to succinctly provide information needed by the Standing Committee on Research (SCOR) and other national decision-makers responsible for requesting or approving research program funding levels during federal transportation program reauthorization deliberations. A user may customize the default report format by eliminating rows or columns of information. The title of the report may also be customized. An authorized user may save the created report to their computer. The Research Program Manager may save a customized report configuration on RPM-Web to make it available to other system users within their agency authorized to view this report.

G- 19 Performance Measure Listing The thirty research performance measures include all of those reported to be commonly used by state transportation agencies in 2004. As discussed in Section 2.0, these performance measures are each assigned to one of five categories. The short names for all performance measures are listed by category. Click on the short name to obtain the full name, definition and additional information about the performance measure. You may also browse information on all included performance measures by clicking View All PM Tables.

Next: Appendix H PM Selection Wizard Questions, Answer Weights, and Comments »
Performance Measurement Tool Box and Reporting System for Research Programs and Projects Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s National Cooperative Highway Research Program (NCHRP) Web-Only Document 127: Performance Measurement Tool Box and Reporting System for Research Programs and Projects explores the integration of standard performance measures and tools to assist users in implementing performance measures into the Research Performance Measurement (RPM) System.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!