Performance Measurement Tool Box and Reporting System for Research Programs and Projects (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

19 CHAPTER 4 – TOOLS FOR RESEARCH PERFORMANCE MEASUREMENT The RPM System includes a variety of tools to assist research program managers and other system users in establishing and conducting performance measurement. A number of tools are available in both RPM-Tools and RPM-Web. Exceptions are noted in the following descriptions. PM 101 PM 101 is a narrated tutorial available only in RPM-Tools. This tutorial is an introduction to performance measurement principles and the application of these principles to research program activities. It also provides an orientation to each of the tools included in the RPM System. The tutorial is composed of the following sections: • RPM System Overview, • What Is Performance Measurement?, • What Makes a Good Performance Measure?, • How Does Research Performance Measurement Help the Research Program Manager?, • What Research Performance Measures Are Commonly Used?, • What Tools Are in the RPM System?, • How Do I Get Started?, • Selecting Performance Goals, • Entering Information into the RPM System, • Measuring Performance, • Creating Performance Reports, and • Performance Measures Listing. A comprehensive, narrated tutorial was not originally envisioned to be part of the tool box to be developed. However, during a meeting early in the project between the panel members and research team, it became recognized that many RPM System users might benefit greatly from an educational module in the tool box which covered the basics of performance measurement and performance management as they apply to research operations. It was also recognized that a description of the tools included in the tool box would be necessary. The research team developed PM 101 as a narrated tutorial to address these needs. The narration text is included as Appendix G. One of the most important sections of PM 101 discusses the value of performance measurement. This section helps the user understand how performance measurement can be an effective, practical tool in managing a state transportation research program. In addition to providing an overview of performance measurement and management in general, PM 101 walks the user through the available tools within the RPM System as well as a step-by-step method to get the user started using the system. At the heart of PM 101 is a detailed explanation of the performance measures which have been included in the RPM System. The 30 standard performance measures which are included in the RPM System are defined and described in detail as part of PM 101. The following information is included in the Performance Measure Listing section of PM 101. The RPM-Tools screen allowing immediate access to information about any of the included performance measures is shown in Figure 5.

20 Figure 5. RPM-Tools - Standard Performance Measure Definitions Home Screen Clicking on any performance measure on this screen will provide the following information about that measure: • definition; • type of measure (outcome, output, resource allocation, efficiency, or stakeholder); • common inputs needed to measure the performance attribute; • formula for calculating performance; • reasons to use the performance measure; • challenges with use of the performance measure; and • typical target audience for the performance measure. PM 101 is a unique tool which serves to educate and motivate the user in an area of management which often seems overwhelming and impractical. In order to maximize the learning environment, the research team employed Bloom’s taxonomy, which is a common approach for developing curriculum. PM 101 was designed to provide the user with the opportunity to acquire competency in the area of performance measurement at several levels: knowledge, comprehension, application, analysis, synthesis, and evaluation. By taking this approach, the team ensured that the user could initially capture basic knowledge and then could acquire additional competencies or needed information as they were working in other

21 areas of RPM-Tools. PM 101 is easy to access while the user is working in the other parts of RPM-Tools. PM SELECTION WIZARD The PM Selection Wizard is only available in RPM-Tools. It is an analysis tool that offers guidance during the process of selecting research performance measures for an agency. The wizard is composed of 10 multiple choice questions which, when answered, attribute various point weights to each of the 30 standard performance measures included within the RPM System. The questions, optional answers, and point weighting system are shown in Appendix H. After all questions are answered, the wizard totals the points attributed to each performance measure and, based on these totals, reports each performance measure as strongly recommended, recommended, or as an alternative to be considered for the user’s agency. The user then makes an initial selection of performance measures for the organization. The user may consider wizard recommendations but is not constrained in making their selections in any way. The Wizard Recommendations & PM Selection Worksheet screen where the user makes selections is shown as Figure 6. Figure 6. RPM-Tools - Wizard Recommendation & PM Selection Worksheet Screen

22 Once the user has made a selection of performance measures in the right-hand column, the selections are displayed as a set and the wizard provides comments appropriate to their set of performance measures. The manner in which the selected set and comments are displayed is shown in Figure 7. The user may then modify the set that has been selected if desired. The wizard questions, the optional answers and associated point weights, and the set of programmed comments which appear when the set of performance measures warrants are based on the opinion of the research team. Figure 7. RPM-Tools – Selected Performance Measures with Wizard Commentary RESOURCE COLLECTION The Resource Collection is a listing of sources for statistics and other information frequently needed during the process of estimating benefits to be derived from research products. The listing is composed primarily of information sources which may be found on web sites, and URLs are provided for quick access. The Resource Collection contains 87 information sources. These information sources are listed in Appendix I. The resources are categorized by topic as shown in Table 4 to facilitate browsing in both RPM-Tools and RPM-Web. A search capability is also provided with the collection in RPM-Web.

23 Table 4. Resource Collection Topics and Numbers of References Provided Resource Topic Number of Resources Included Economics and Finance 40 Energy and Environment 10 Freight Transportation 14 Infrastructure 18 Passenger Travel 11 Registered Vehicles and Vehicle Miles Traveled 0 Safety 26 BENEFIT ESTIMATION WORK SHEETS At the product level, users have the opportunity to estimate benefits using different methodologies, each with associated work sheets to assist the user through the process. The intent of these options is to provide the user as much flexibility as possible to address the wide variety of research products which exist. Automated work sheets are available in the RPM System for each methodology. Accessing these benefit estimation work sheets within RPM-Web and initiating benefit estimation for a research product require minimal preliminary information entry. If the research project title, start date, and end date are already available in the database, as will be the case if the user has previously entered them into the RiP database, accessing the benefit estimation work sheets requires only that the user enter the name of the research product and check the appropriate box to indicate that the product is or will be implemented. At the heart of every benefit estimation is the listing of pertinent statistical data and the assumptions which are involved in the estimation. The Resource Collection described above is one source for necessary statistical information. Assumptions that are involved should be obtained from the most knowledgeable individuals within the agency. It is critical to estimation credibility that sources of statistical data and assumptions are documented in the work sheets provided to the user. RPM-Web offers the user the three methodologies listed below for estimating benefits. RPM-Tools offers the first method. • Current Minus Future Method. This method requires two determinations of costs, fatalities, and/or numbers of crashes. While this method is almost universally applicable to benefit determination situations, it usually requires more statistical data than the other two methods described below. The user is

24 first walked through a series of worksheets which establish the currently experienced annual costs, fatalities, and/or numbers of crashes associated with the situation to be improved by the research product. Then, the user is led through similar worksheets to establish the expected annual costs, fatalities, and/or numbers of crashes after agency-wide implementation of the research product. The difference between the two determinations provides an estimate of annual benefits, which is the basis of the total benefits determination calculation embedded in the program. The length of time estimated to achieve agency-wide implementation and the estimated useful life of the research product are other important factors in the determination of total expected benefits. • Direct Difference Method. This method is particularly well suited for use when the research project provides estimated benefits per application of the research product, or when the expected benefits per application can be estimated after the research project is completed. This method is simpler than the current minus future method because it does not require determinations of agency-wide costs, fatalities, and/or numbers of crashes. Instead, agency-wide annual benefits are estimated by multiplying the expected benefits from each application of the research product by the number of locations or applications where the product will be implemented. • Percent Improvement Method. This method is ideal when the research project determines a percentage improvement to be expected in costs, fatalities, and/or numbers of crashes, or when a percentage improvement can be estimated after the research project is completed. This method requires the estimator to determine the current annual costs, fatalities, and/or number of crashes associated with the situation to be improved by the research product. Then, the percentage improvement is applied to determine annual expected benefits. The work sheets of each type lead the user through the process of entering necessary information about the product, entering data and information used in the benefit estimation, and documenting the sources for the data and information being used. CATALOG OF BENEFIT ESTIMATION EXAMPLES The Benefit Estimation Catalog provided in both RPM-Tools and RPM-Web was developed primarily from actual estimates of benefits performed and submitted to the research team by state transportation agencies. In addition, several benefit estimations in the catalog of examples were developed for hypothetical products so that a wider variety of examples could be included. The hypothetical products and benefit estimation examples are clearly indicated as such within the catalog. The catalog of example benefit estimations is provided so that the RPM System user may find assistance in determining how to undertake estimating benefits to be obtained from a product at hand. All three of the methods for determining benefits are included among the benefit estimation examples in the catalog. The RPM-Web screen containing the list of example benefit determinations is shown in Figure 8. One of the 20 standard example estimates is shown in Figure 9. All standard example estimates are included in Appendix J. These examples demonstrate viable approaches to estimating benefits for a variety of types of research products. Should an example estimation be found which is similar in nature to the research product and benefits

25 to be estimated by the user, work sheets which contain the information from the example may be accessed and modified as necessary to develop the desired product benefit estimation. Whether a blank work sheet is used or an example is being obtained from the catalog, the user is provided automated work sheets which both assist in the calculations involved and in documenting how the estimation was performed. Figure 8. Benefit Estimate Example Catalog

26 Figure 9. Benefit Estimation Example

27 AUTOMATED PRESENT VALUE CALCULATION If the RPM System user enters a discount rate to be used in the dollar cost savings estimation, the system provides automated conversion of research project related costs and cost savings to their present values for the year the research project was completed. Considering the time value of dollars invested in research is important because the period of time from initiating a research project until the implemented products have become obsolete is often fifteen years or longer. Lengths of time of this order make present value handling of dollar values a necessity. Failure to address the time value of invested capital may become a serious credibility issue for consumers of the information being provided. The discount rate is a percentage used to convert annual dollar costs or benefits occurring in future or earlier years to their values during the desired comparison year. While discount rates used by private businesses usually include the minimum acceptable profit, established by company owners, it is recommended that public agencies’ cost to borrow money, or “cost of capital,” be used as their discount rate (5). These rates have historically ranged from three percent to five percent (6). This percentage, or a similarly determined percentage, should be obtained by inquiring with the finance office of the state transportation agency performing the benefits estimate. Useful information about application of discount rates may also be found on the web site for the Office of Management and Budget (OMB). The benefit estimation work sheets of the RPM System consider and automatically convert three types of dollar transactions into present value quantities. These are the annual research project costs, the annual implementation costs, and the agency’s annual cost savings which begin to be realized at the time implementation is initiated. As is customary in economic analyses, the entire cost or cost savings occurring throughout a year is considered transacted on the last day of that year. The cash flow diagram depicted in Figure 10 shows these cash events occurring over time. 1995 1998 2001 2004 2010 Annual Research Project Costs Total Period of Agency Cost Savings Implementation Costs Period of Phased-in Implementation Annual Agency Cost Savings Active Research Estimated Year Product Becomes Obsolete Figure 10. Research Project and Benefit Cash Flow Diagram

28 Please note several things in the Figure 10 example. First, the research project shown was completed in 1998. So the discount value will convert all annual dollar costs occurring either before or after 1998 to the value of those dollars in 1998. Likewise, the annual cost savings occurring after project termination will be discounted to 1998 value. Note also that in this example that active use of the product did not begin until a year after completion of the research project. While this may not always be the case, periods of product evaluation often do require a year or more. The implementation period shown in Figure 10 to obtain a statewide, maximum- feasible implementation level for the agency is three years. The RPM System handles all multi-year implementation periods using a straight-line ramped increase for the cost savings over this time period. For implementation costs, the RPM System totals all implementation costs and then averages them and distributes these averages over the years of the implementation period. A ramped distribution is not used because of the greater likelihood of substantial up-front costs to initiate implementation. The decision to ramp and average these two transition period costs was made to simplify calculations in the prototype RPM System. To perform a quantitative estimation of benefits expected from a research product, it is necessary to estimate the length of time that will transpire before the product becomes obsolete and is replaced by later innovations. The anticipated service life of a product before obsolescence is an entry item for the user during benefit estimation. To assist the user in selecting a predicted service life, guidance is provided within the system for a variety of general types of research products. The guidance provided was developed by surveying AASHTO RAC members during the 2004 national RAC meeting in Mystic, Connecticut. Those attending this meeting were provided an opportunity to fill in a form wherein they indicated their opinion on average useful life for 22 different generic research product types. The responses obtained and the guidance being provided within the RPM System derived from them are shown in Appendix K. PERFORMANCE MEASUREMENT REPORTS A suite of pre-programmed yet customizable reports is provided in RPM-Web. Performance report types provided include the Individual Project Report, the Multiple Project Report, the Research Program Report, and both a State and a National Research Program Summary Report. In addition, a Security Report is provided to assist the research program manager administer access to the agency’s records in RPM-Web. An Individual Product Benefit Estimation Report is available from RPM-Tools. Customization made available for the RPM-Web reports includes deleting or adding performance measures to create any combination desired. The user may also add the agency’s unique performance measures as well as goals and performances for these additional performance measures. Additionally, the user may rename any of the reports and may add a subtitle to the reports. Finally, any of the columns displayed may be deleted from the report being created, if desired. When a desired report format has been created, the user has the option to save that format for later use. An example of an Individual Project Report in its RPM-Web view is shown in Figure 11. Note that multiple sponsors, when applicable, are displayed along with individual sponsor performance measurement information. When quantitative benefit estimations have been made, those calculations are also displayed as follow-on information to this report.

29 Figure 11. RPM-Web – Individual Project Report Screen

30 An example of a Research Program Report in its RPM-Web view is shown in Figure 12. All performance measures are displayed in the Figure 12 example although developers of these reports have the option to show only selected performance measures. The PDF format of the effectiveness portion of the Research Program Report is shown in Figure 13. Each of the RPM-Web reports has a PDF format option. Guidance in how to generate RPM System reports is provided in the PM 101 tutorial. Although the user can customize all performance measurement reports provided by the RPM System, the standard report formats have been designed to provide an easy alternative for summarizing research performance measurement information.

31 Figure 12. RPM-Web - State Program Report Screen

32 Figure 13. RPM-Web - Research Program Effectiveness Portion of the State Program Report in PDF Format