Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
ACKNOWLEDGMENT This work was sponsored by the American Association of State Highway and Transportation Officials (AASHTO), in cooperation with the Federal Highway Administration, and was conducted in the National Cooperative Highway Research Program (NCHRP), which is administered by the Transportation Research Board (TRB) of the National Academies. COPYRIGHT PERMISSION Authors herein are responsible for the authenticity of their materials and for obtaining written permissions from publishers or persons who own the copyright to any previously published or copyrighted material used herein. Cooperative Research Programs (CRP) grants permission to reproduce material in this publication for classroom and not-for-profit purposes. Permission is given with the understanding that none of the material will be used to imply TRB, AASHTO, FAA, FHWA, FMCSA, FTA, Transit Development Corporation, or AOC endorsement of a particular product, method, or practice. It is expected that those reproducing the material in this document for educational and not-for- profit uses will give appropriate acknowledgment of the source of any reprinted or reproduced material. For other uses of the material, request permission from CRP. DISCLAIMER The opinion and conclusions expressed or implied in the report are those of the research agency. They are not necessarily those of the TRB, the National Research Council, AASHTO, or the U.S. Government. This report has not been edited by TRB.
iv ACKNOWLEDGMENTS The research reported herein was performed under NCHRP Project 20-63 by the Texas Transportation Institute (TTI), a member of the Texas A&M University System. Paul E. Krugler, P.E., research engineer and manager of implementation, was the principal investigator. He is a retired employee of the Texas Department of Transportation, having served his last five years with that agency as their research program director. Melissa Noggle Walden, P.E., associate research scientist, served as the co-principal investigator. The work was done under the general supervision of Paul Krugler. Other authors of the report include Melissa Walden; Bradley Hoover, lead software applications developer; Yu-ying Diana Lin, systems analyst; and Sandra Tucker, who was the TTI library services manager and is now associate professor at the Texas A&M University Libraries. The research team would like to gratefully acknowledge the extraordinary level of assistance and guidance provided by the project panel throughout this project. In particular, Richard Long should be recognized. As the panel chairman and also vice chair of the American Association of State Highway and Transportation Officials (AASHTO) Research Advisory Committee (RAC), it was his vision and unrelenting efforts which made this project become reality. The coordination of the database developed during this project with the Research in Progress database would not have been possible without the greatly appreciated support and assistance of the National Academy of Science. Finally, the numerous AASHTO RAC members that readily shared their experiences in research performance measurement, including a number of excellent quantitative benefit estimates, were most instrumental in this project and should also be warmly thanked.
v NCHRP PROJECT 20-63 PANEL Area of Special Projects Richard C. Long, Florida DOT, Tallahassee, FL (Chair) Elizabeth G. Bonini, Pennsylvania DOT, Harrisburg, PA Rick Collins, Texas DOT, Austin, TX O. A. Elrahman, New York State DOT, Watervliet, NY Monique R. Evans, Ohio DOT, Columbus, OH David L. Huft, South Dakota DOT, Pierre, SD Russell Lee, Oak Ridge National Laboratory, Oak Ridge, TN Richard L. McReynolds, Kansas DOT, Topeka, KS Sue Sillick, Montana DOT, Helena, MT Jeffrey H. Smith, Maryland State Highway Administration, Baltimore, MD Loren L. Turner, California DOT, Sacramento, CA Joe Parks, FHWA Liaison Mark R. Norman, TRB Liaison
vi ABSTRACT This report documents the work performed to gather and analyze available research performance measurement information, select a balanced and broadly applicable set of these performance measures, develop tools to assist practitioners in applying these measures to their research projects and programs, and deliver these products to the community of state research program managers. The selected performance measures and the developed tools were integrated to create the Research Performance Measurement (RPM) System, composed of a web site, RPM-Web, and a complementing CD-ROM tool box, RPM-Tools. The tool set being provided within the system includes PM 101, a narrated research performance measurement tutorial; a wizard to assist in selecting research program performance measures; a compendium of data resource links; a catalog of example research benefit estimations; and automated work sheets for the practitioner to create new benefit estimations. Additional functionalities provided in RPM-Web include historical performance information storage and the capability to generate a suite of performance reports from database information.
vii TABLE OF CONTENTS LIST OF FIGURES .................................................................................................................. ix LIST OF TABLES ......................................................................................................................x SUMMARY OF FINDINGS ......................................................................................................1 CHAPTER 1 - INTRODUCTION AND RESEARCH APPROACH ...................................2 Introduction..........................................................................................................................2 Research Approach ..............................................................................................................3 CHAPTER 2 â INFORMATION GATHERING AND ANALYSIS ....................................7 Literature Search..................................................................................................................7 National Surveys..................................................................................................................8 CHAPTER 3 â RESEARCH PERFORMANCE MEASURES............................................13 CHAPTER 4 â TOOLS FOR RESEARCH PERFORMANCE MEASUREMENT..........19 PM 101...............................................................................................................................19 PM Selection Wizard .........................................................................................................21 Resource Collection...........................................................................................................22 Benefit Estimation Work Sheets........................................................................................23 Catalog of Benefit Estimation Examples...........................................................................24 Automated Present Value Calculation ...............................................................................27 Performance Measurement Reports...................................................................................28 CHAPTER 5 â CONCLUSIONS AND RECOMMENDATIONS .......................................33 Conclusions........................................................................................................................33 Recommendations to RPM System Users .........................................................................33 Suggested Research ...........................................................................................................34 REFERENCES .........................................................................................................................35 BIBLIOGRAPHY.....................................................................................................................36 APPENDIX A Glossary of Terms ................................................................................... A-1 APPENDIX B Systems Requirements Document Excerpts ...........................................B-1 APPENDIX C Survey Instruments .................................................................................. C-1 APPENDIX D Organizations Responding to Survey ..................................................... D-1 APPENDIX E RAC Survey Performance Measure Comments .....................................E-1 APPENDIX F Comprehensive List of Performance Measures ......................................F-1 APPENDIX G Performance Measurement 101 Narration............................................ G-1 APPENDIX H PM Selection Wizard Questions, Answer Weights, and Comments ... H-1
viii APPENDIX I Resource Collection Items..........................................................................I-1 APPENDIX J Standard Benefit Estimation Examples .................................................. J-1 APPENDIX K Useful Product Life Guidance................................................................. K-1
ix LIST OF FIGURES Figure 1. TRIS, RiP and RPM Usage Timeline........................................................................ 2 Figure 2. RPM-Web - Navigation Story Board ........................................................................ 4 Figure 3. RPM-Tools - Welcome Screen.................................................................................. 5 Figure 4. RPM-Web - Welcome Screen ................................................................................... 5 Figure 5. RPM-Tools - Standard Performance Measure Definitions Home Screen............... 20 Figure 6. RPM-Tools - Wizard Recommendation & PM Selection Worksheet Screen......... 21 Figure 7. RPM-Tools â Selected Performance Measures with Wizard Commentary ............ 22 Figure 8. Benefit Estimate Example Catalog.......................................................................... 25 Figure 9. Benefit Estimation Example.................................................................................... 26 Figure 10. Research Project and Benefit Cash Flow Diagram ............................................... 27 Figure 11. RPM-Web - Individual Project Report Screen...................................................... 43 Figure 12. RPM-Web - State Program Report Screen............................................................ 31 Figure 13. RPM-Web - Research Program Effectiveness Portion of the State Program Report in PDF Format.......................................................................................................... 32
x LIST OF TABLES Table 1. Mean Perceived-Value Ratings for Performance Measures, by Survey Group ....... 10 Table 2. Comparison of Performance Measure Rankings, by Survey Group......................... 12 Table 3. Standard Research Performance Measures Included in the RPM System................ 14 Table 4. Resource Collection Topics and Numbers of References Provided ......................... 23