Designing the Archive for SHRP 2 Reliability and Reliability-Related Data (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

TRANSPORTATION RESEARCH BOARD WASHINGTON, D.C. 2015 www.TRB.org The Second S T R A T E G I C H I G H W A Y R E S E A R C H P R O G R A M REPORT S2-L13A-RW-1 Designing the Archive for SHRP 2 Reliability and Reliability-Related Data RobeRt HRanac Iteris, Inc. Berkeley, California JoRge a. baRRios Kittelson & Associates, Inc. Oakland, California

Subject Areas Highways Data and Information Technology Operations and Traffic Management Safety and Human Factors

The Second Strategic Highway Research Program America’s highway system is critical to meeting the mobility and economic needs of local communities, regions, and the nation. Developments in research and technology—such as advanced materials, communications technology, new data collection tech- nologies, and human factors science—offer a new opportunity to improve the safety and reliability of this important national resource. Breakthrough resolution of significant transportation problems, however, requires concentrated resources over a short time frame. Reflecting this need, the second Strategic Highway Research Program (SHRP 2) has an intense, large-scale focus, integrates multiple fields of research and technology, and is fundamentally different from the broad, mission-oriented, discipline-based research programs that have been the mainstay of the highway research industry for half a century. The need for SHRP 2 was identified in TRB Special Report 260: Strategic Highway Research: Saving Lives, Reducing Congestion, Improving Quality of Life, published in 2001 and based on a study sponsored by Congress through the Transportation Equity Act for the 21st Century (TEA-21). SHRP 2, modeled after the first Strategic Highway Research Program, is a focused, time- constrained, management-driven program designed to com- plement existing highway research programs. SHRP 2 focuses on applied research in four areas: Safety, to prevent or reduce the severity of highway crashes by understanding driver behavior; Renewal, to address the aging infrastructure through rapid design and construction methods that cause minimal disruptions and produce lasting facilities; Reliability, to reduce congestion through incident reduction, management, response, and mitigation; and Capacity, to integrate mobility, economic, environmental, and community needs in the planning and designing of new trans- portation capacity. SHRP 2 was authorized in August 2005 as part of the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU). The program is managed by the Transportation Research Board (TRB) on behalf of the National Research Council (NRC). SHRP 2 is conducted under a memo- randum of understanding among the American Association of State Highway and Transportation Officials (AASHTO), the Federal Highway Administration (FHWA), and the National Academy of Sciences, parent organization of TRB and NRC. The program provides for competitive, merit-based selection of research contractors; independent research project oversight; and dissemination of research results. SHRP 2 Reports Available by subscription and through the TRB online bookstore: www.mytrb.org/store Contact the TRB Business Office: 202-334-3213 More information about SHRP 2: www.TRB.org/SHRP2 SHRP 2 Report S2-L13A-RW-1 ISBN: 978-0-309-27431-9 © 2015 National Academy of Sciences. All rights reserved. Copyright Information Authors herein are responsible for the authenticity of their materials and for obtaining written permissions from publishers or persons who own the copy- right to any previously published or copyrighted material used herein. The second Strategic Highway Research Program grants permission to repro- duce material in this publication for classroom and not-for-profit purposes. Permission is given with the understanding that none of the material will be used to imply TRB, AASHTO, or FHWA endorsement of a particular prod- uct, method, or practice. It is expected that those reproducing material in this document for educational and not-for-profit purposes will give appropriate acknowledgment of the source of any reprinted or reproduced material. For other uses of the material, request permission from SHRP 2. Note: SHRP 2 report numbers convey the program, focus area, project number, and publication format. Report numbers ending in “w” are published as web documents only. Notice The project that is the subject of this report was a part of the second Strategic Highway Research Program, conducted by the Transportation Research Board with the approval of the Governing Board of the National Research Council. The members of the technical committee selected to monitor this project and review this report were chosen for their special competencies and with regard for appropriate balance. The report was reviewed by the technical committee and accepted for publication according to procedures established and overseen by the Transportation Research Board and approved by the Governing Board of the National Research Council. The opinions and conclusions expressed or implied in this report are those of the researchers who performed the research and are not necessarily those of the Transportation Research Board, the National Research Council, or the program sponsors. The Transportation Research Board of the National Academies, the National Research Council, and the sponsors of the second Strategic Highway Research Program do not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the object of the report.

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. On the authority of the charter granted to it by Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achieve- ments of engineers. Dr. C. D. (Dan) Mote, Jr., is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, on its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. (Dan) Mote, Jr., are chair and vice chair, respectively, of the National Research Council. The Transportation Research Board is one of six major divisions of the National Research Council. The mission of the Transportation Research Board is to provide leadership in transportation innovation and progress through research and information exchange, conducted within a setting that is objective, interdisci- plinary, and multimodal. The Board’s varied activities annually engage about 7,000 engineers, scientists, and other transportation researchers and practitioners from the public and private sectors and academia, all of whom contribute their expertise in the public interest. The program is supported by state transportation departments, federal agencies including the component administrations of the U.S. Department of Transporta- tion, and other organizations and individuals interested in the development of transportation. www.TRB.org www.national-academies.org

ACKNOWLEDGMENTS This work was sponsored by the Federal Highway Administration in cooperation with the American Asso- ciation of State Highway and Transportation Officials. It was conducted in the second Strategic Highway Research Program, which is administered by the Transportation Research Board of the National Academies. This project was managed by William Hyman, Senior Program Officer for SHRP 2 Reliability. The research reported on herein was performed by Iteris, Inc., supported by Kittelson & Associates, Inc. Robert Hranac, Ali Mortazavi, and Karl Petty of Iteris, Inc., were the principal investigators. The authors acknowledge the contributions to this research and final report from Michael Darter, Sandra Lennie, Leon Raykin, Kavya Sambana, Michael Walls, Dustin Salentiny, Ken Yang, and Alex Kurzhanskiy of Iteris, Inc., and Jorge Andres Barrios, Alex Skabardonis, Senanu Ashiabor, Rick Dowling, and Daphne Dethier of Kittelson & Associates, Inc. The L13A project team would like to acknowledge all the principal investigators of the SHRP 2 Reliability projects, the members of the SHRP 2 Reliability Technical Coordinating Committee, and the L13A Technical Expert Task Group. SHRP 2 STAFF Ann M. Brach, Director Stephen J. Andrle, Deputy Director Cynthia Allen, Editor Kenneth Campbell, Chief Program Officer, Safety Jared Cazel, Editorial Assistant JoAnn Coleman, Senior Program Assistant, Capacity and Reliability Eduardo Cusicanqui, Financial Officer Richard Deering, Special Consultant, Safety Data Phase 1 Planning Shantia Douglas, Senior Financial Assistant Charles Fay, Senior Program Officer, Safety Carol Ford, Senior Program Assistant, Renewal and Safety James Hedlund, Special Consultant, Safety Coordination Alyssa Hernandez, Reports Coordinator Ralph Hessian, Special Consultant, Capacity and Reliability Andy Horosko, Special Consultant, Safety Field Data Collection William Hyman, Senior Program Officer, Reliability Linda Mason, Communications Officer David Plazak, Senior Program Officer, Capacity and Reliability Rachel Taylor, Senior Editorial Assistant Dean Trackman, Managing Editor Connie Woldu, Administrative Coordinator

The Reliability Technical Coordinating Committee recognized early in the formation of the Reliability research program that there would be a significant benefit to the research community, traffic engineers, planners, data managers, and others if the data from all the Reliability-related research projects could be preserved and easily accessed over the next 25 years or more. The inspiration for a repository was the Long Term Pavement Perfor- mance monitoring system from SHRP 1. Because it was not clear how to proceed, the TCC determined that first there should be a feasibility study, and if that study found that it was both possible and desirable to develop the Reliability Archive, the Archive would be built. At the outset, the TCC also set aside resources to provide support to enable contractors to populate the Archive with data. The feasibility study strongly suggested that developing the Archive was both desirable and fea- sible. The study called for the use of open source software and determined that both the software to realize the functional capability of the Archive and the data should be stored in the cloud. The TCC recommended that the Archive be developed. Although the priority from the outset was to make data sets used in the SHRP 2 Reliability research available for many decades, it quickly became apparent that the data would not be understandable or useful without sufficient contextual information. Thus, it was deter- mined that all types of data should go into the Archive, both structured and unstructured. Structured data include comma-separated data, other flat files, and relational data. Unstruc- tured data include data dictionaries, reports, presentations, video, spreadsheets, computer code, and other digital objects. In the parlance of the Archive, structured data are called “Data sets” and unstructured data, “Non–data sets.” The Archive’s home page shows the five main use cases: upload, search, visualize, download, and discuss. Also shown on the home page are different statistics about the Archive and the latest four artifacts that have been uploaded so a user can drill down into the most recent additions. “Artifact” is the term used for each data set and non–data set in the Archive. There are three ways to view the data once in the Archive: (1) a word search; (2) a search of the Archive that shows data sets by geographical location on a map of the country; and (3) a search by SHRP 2 focus area and project listing. The word search is a simple text search. A “search of the Archive” produces a literal spiral of data sets at each location; the user can click on each one and drill down to explore what the data are. A user can also select whether to look at a data set or non–data set and provide further filters for the selection. For example, a user can filter a data set according to whether it has speed, occupancy, and flow data. Finally, a user can search by project by first clicking on a focus area, then click- ing a project listed under that focus area. An informative description accompanies each project to acquaint the user with the project objectives, key considerations in undertaking the research, and some of the important research products produced. Every project has metadata, and the metadata for data sets includes a data dictionary. F O R EWO R D William Hyman, SHRP 2 Senior Program Officer, Reliability

The Archive has substantial visualization capability for a user to preview and evaluate whether a data set is of interest. This feature is only for data sets, not non–data sets. This capability offers three different types of visualization. A user can visualize the first 300 records of a data set. On a map of highway facilities, reminiscent of a GPS navigation map, a user can see the precise location of the traffic detectors used to collect the data. Finally, a user can graph the relationships between different numerical fields in the data set. An example might be a scatter plot of the relationship between speed and flow. The Archive was developed with the flexibility to be a dynamic and living system so that research that is new or related to the original Reliability research could be added to the sys- tem. There are significant administrative, maintenance, and operations costs to this capa- bility. As of the end of the Archive contract, only SHRP 2 Reliability-related data had been entered into the system.

C O N T E N T S 1 Executive Summary 4 CHAPTER 1 Background 4 1.1 Objective 4 1.2 Target Audience 4 1.3 Benefits 4 1.4 Summary of Archived Projects and Artifacts 5 1.5 Document Organization 8 CHAPTER 2 Approach 8 2.1 General Approach 9 2.2 Software Development Methodology 10 2.3 User Engagement 13 CHAPTER 3 Preparatory Analysis 13 3.1 Review of the L13 Report 18 3.2 Archived Data User Services 30 3.3 Online Archiving Systems 35 3.4 Commercially Available Archiving Technologies 37 3.5 Summary of the Preparatory Analysis 39 CHAPTER 4 System and User Needs and Requirements 39 4.1 System Overview 39 4.2 Roles 41 4.3 System Needs 43 4.4 High-Level System Features 45 4.5 Scenarios 49 CHAPTER 5 Artifact Upload 49 5.1 Types of Artifacts Archived 49 5.2 Artifact Ingestion Process 52 5.3 Data Dictionary 53 5.4 Metadata 57 5.5 Artifact Relationships 57 5.6 Preparing Artifacts for Upload 59 5.7 Supplementary Documents to Assist Principal Investigators and Creators 60 5.8 Quality Assurance After Uploading Artifacts 61 CHAPTER 6 User Guide—Working with the Archive 61 6.1 Creating and Managing Your User Account 61 6.2 Remove an Account 63 6.3 Search for Artifacts 64 6.4 Working with Artifacts 72 6.5 Download an Artifact

73 6.6 Discussion 73 6.7 Uploading Artifacts 76 6.8 Automatically Generated E-mail Notifications 79 CHAPTER 7 System High-Level Architecture 79 7.1 Amazon Web Services 79 7.2 WordPress 81 7.3 MySQL Database 81 7.4 Solr Search Engine Server 81 7.5 S2A Server 87 CHAPTER 8 Test Plan 87 8.1 Development Testing Strategy 88 8.2 System Tests 92 8.3 List of Artifacts Needed to Run the Test Plan 93 CHAPTER 9 Notes on Operations and Maintenance of the Archive 93 9.1 Inclusion of User-Submitted Data 94 9.2 Key Issues Associated with Operations and Maintenance 99 9.3 Managing Issues with Non–SHRP 2 Data 104 References 106 Appendix A. Data Dictionary Template 107 Appendix B. Federal System Security Guidelines