National Academies Press: OpenBook

Application of Big Data Approaches for Traffic Incident Management (2023)

Chapter: Chapter 7 - Conclusions and Recommendations

« Previous: Chapter 6 - TIM Big Data Guidelines
Page 89
Suggested Citation:"Chapter 7 - Conclusions and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2023. Application of Big Data Approaches for Traffic Incident Management. Washington, DC: The National Academies Press. doi: 10.17226/27300.
×
Page 89
Page 90
Suggested Citation:"Chapter 7 - Conclusions and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2023. Application of Big Data Approaches for Traffic Incident Management. Washington, DC: The National Academies Press. doi: 10.17226/27300.
×
Page 90
Page 91
Suggested Citation:"Chapter 7 - Conclusions and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2023. Application of Big Data Approaches for Traffic Incident Management. Washington, DC: The National Academies Press. doi: 10.17226/27300.
×
Page 91
Page 92
Suggested Citation:"Chapter 7 - Conclusions and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2023. Application of Big Data Approaches for Traffic Incident Management. Washington, DC: The National Academies Press. doi: 10.17226/27300.
×
Page 92

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

89   NCHRP Project 03-138 sought to demonstrate the feasibility and practical value of big data approaches for TIM and to provide guidelines to address the findings and recommenda- tions of NCHRP Research Report 904 (Pecheux, Pecheux, & Carrick, 2019). To accomplish these objectives, the research team conducted interviews with TIM programs to identify use cases for the project, performed a comprehensive assessment of traditional and big data sources relevant to the TIM use cases, established four big data pipelines and associated data products, and created guidelines that expand and refine those from NCHRP Research Report 904. This chapter provides conclusions based on the research findings and offers transportation agencies potential next steps to ready the data and to advance their capability maturity by capitalizing on traditional and emerging data sources and implementing big data approaches to improve TIM. 7.1 Conclusions TIM programs, implemented by transportation agencies and their responder agency partners, collect a wide range of data, but most of these data are not in the realm of big data. Furthermore, given that separate organizations (i.e., transportation, law enforcement, fire, towing, EMS) produce these data, most are used only in isolation, if at all. If these disparate data were brought together, they could be leveraged in modern ways to gain insights into TIM practices, policies, and performance. Discussions with transportation agencies as part of this research project revealed TIM pain points that a data-driven, big data approach might be able to address. However, a comprehensive assessment of relevant TIM data sources showed that, in addition to existing data silos, the data themselves are also a barrier to using and implementing big data approaches for TIM. Lack of access to data due to traditional policies and practices, privacy concerns, and inadequate data systems; deficient data quality; and lack of standardization of data within and across agencies hinder the use of TIM data in big data pipelines. Additionally, data are not the transportation agencies and TIM programs’ only barrier to capitalizing on emerging data sources. Big data and modern data management go hand in hand, as modern data management practices, tools, and techniques are required to handle the massive amounts of structured and unstructured data produced by emerging technologies. Most trans- portation agencies, however, are not yet equipped to store, manage, integrate, and analyze these big data. The research team, with input from multiple TIM programs, identified four use cases with which to demonstrate the feasibility and practical value of big data approaches for TIM. For each use case, the team gathered and assessed relevant data, then created big data pipe- lines and associated data products, applying the big data concepts and guidelines presented C H A P T E R 7 Conclusions and Recommendations

90 Application of Big Data Approaches for Traffic Incident Management in NCHRP Research Report 904. This process validated the challenges and limitations in imple- menting a big data approach for TIM. The cost and data use agreements associated with third-party data limited the research team to data that were mostly available free of charge. As such, navigation app data, provided free to the states as part of a unique partnership with public entities, was a valuable source of big data in two of the use cases. The team purchased a small window of data from a third-party CV pro- vider for one of the use cases and inexpensive third-party weather data for two of the use cases. One of the use cases primarily used transportation agency data (i.e., crash data), with a focus on integrating these data across multiple states and enriching them with roadway and weather data to create a more robust set of multistate data for analysis. Development of the data pipelines for each of the four use cases served to test and validate various big data methods and approaches, as well as to identify challenges and limitations in the big data approach for TIM. While the approaches worked, the development of each use case resulted in various challenges and successes. Use Case 1: Improving Incident Detection and Verification to Expedite Response streamed real- time free navigation app data through a simple and inexpensive data pipeline. The data product was a dashboard with functions to visualize and sort ongoing traffic incidents in various ways. Initially, the team developed the pipeline and dashboard using a more resource-intensive database and dashboarding tool. However, a change in the approach eliminated storing the navigation app data and pushing it through to a third-party hosting layer, which simplified the data pipeline while increasing the speed of data processing and decreasing the cost to run the pipeline. Use Case 1 was the most simple and straightforward of the data pipelines developed, and it resulted in a tool that could be used by TMC operators to quickly identify and verify traffic incidents. Use Case 2: Real-Time TIM Timeline and Performance Measures integrated streaming data from navigation app crash alerts, navigation app jam/speed alerts, CAD, and a third-party weather API in real time. This data pipeline was the most complex of the data pipelines developed and presented the most challenges (e.g., memory issues) and limitations (e.g., finding matches between datasets). While the data pipeline was stable for a two-week period, an analysis of the data in the crash document database (i.e., the integrated crash dataset generated by the pipeline) showed few matches across the data sources. Challenges in matching crashes across the datasets stemmed from multiple issues. First, the sample size of the third-party data is relatively small due to the crowdsourced nature of the data. As such, only 291 of the CAD crashes were linked to third-party alerts. Second, there is not always a navigation app jam alert associated with a navigation app crash alert or a CAD event, as not all crashes result in disruptions to traffic flow. Third, the CAD data did not always contain the TIM timestamps; in fact, none of the matches included T5 (RCT). Nonetheless, the team tested and proved the concept of integrating multiple real-time datasets for the TIM timestamps. Improvements in data quality and an increased use of navigation app data could produce more complete results in the future. Use Case 3: Understanding Secondary Crashes combined crash data from 10 states (those that collect information on secondary crashes via statewide crash reports) and enriched these data with detailed roadway data from ARNOLD and weather data from a third-party weather API. The objective of this use case was to combine and enrich disparate data across states to better understand when, where, and why secondary crashes occur. The team experienced numerous challenges and limitations with this use case. One of the biggest challenges was creating a uniform dataset from the disparate crash data. The lack of stan- dardization in the crash data led to major inconsistencies across data elements and attributes,

Conclusions and Recommendations 91 making it challenging to map elements/attributes across the states. A spatial-temporal analysis to identify potential primary crashes resulted in the verification of only 30 percent of the secondary crashes. These results suggest that there are issues with the collection/classification of secondary crashes by law enforcement. Crash data quality was another limitation; missing and unknown values, combined with inconsistencies in the data, led to an inconclusive cluster analysis. While the team was successful in assembling the largest set of secondary-crash data to date for analysis and enriching these data with detailed roadway and weather data, data consistency and quality challenges limited the usefulness of the data in uncovering additional insights and trends. Use Case 4: Exploratory Analysis of Third-Party CV Data for Crash Detection was primarily an exploratory data analysis (EDA) of a third-party CV dataset containing one month of driver event data and vehicle movement data in Phoenix, Arizona. In addition to the EDA, the team devel- oped a data pipeline to compare and match the CV driver event data and vehicle movement data with historical crash data from the same time/space window. The team also developed two dashboards (one for the driver event data matched to crashes and one for the vehicle movement data matched to crashes) to visualize the traffic conditions around any crashes that were matched. While the data pipeline was successful at matching a few crashes with the CV data, there were several challenges that limited the number of possible matches. First, the recorded times and geolocations of crashes are not always accurate, which can make it difficult to accurately match the crash data to the CV data. Second, there were crashes that could not be linked to the CV data, demonstrating that the current market penetration rate of connected vehicles is low. Third, there is no established link between the driver event dataset and the vehicle move- ment dataset (for privacy purposes, vehicle movement data only has speed, and speeds are only available within the driver event data if there is a hard braking or hard accelerating event), and attempts to establish a linkage between the two datasets were unsuccessful. Having these two datasets linked could help identify a relationship between the CV data and crash data. The linkages that were made, and the visualization of matched data on the two dashboards, illustrate the value of CV data to better understand crashes and their impacts on the transportation network. 7.2 Recommended Next Steps Recommendations for potential next steps include further readying the data for big data analytics and advancing the capability maturity of transportation agencies to capitalize on traditional and emerging data sources to implement big data approaches to improve TIM. As has been demonstrated throughout this report, much of the traditional TIM-relevant data (e.g., crash, ATMS, CAD) is not in a state that is ready for big data analysis. If agencies plan to integrate these data with other data sources and use them in big data pipelines, efforts should be made to improve the data’s quality and completeness, accessibility and format (e.g., real-time APIs, machine readable, Apache Parquet, and deployed to a cloud environment), and standardization and consistency (e.g., use standards/specifications and add common data ele- ments across data sources). Agencies should also consider more modern and automated ways to collect TIM-relevant data to improve the timeliness and quality of the data. Beyond agency-owned data, transportation agencies routinely purchase and interface with big data from third-party providers, including crowdsourced data, probe vehicle data, and micro- mobility data. Data from CVs, while not as widely available as those previously mentioned, are rapidly emerging and will continue to grow as more vehicles are equipped. The use of these and other big data sources will play a significant role in driving transportation planning, opera- tions, and safety decisions into the future, including for TIM. As such, transportation agencies should ready their organizations to store, integrate, and analyze these data.

92 Application of Big Data Approaches for Traffic Incident Management To do this, agencies need to adopt modern data management practices, which include not only technical practices (e.g., cloud architecture) but institutional and culture practices as well (e.g., procurement, IT, and data sharing). In addition to the guidelines laid out in this report, NCHRP Research Report 904 and NCHRP Research Report 952 offer guidelines, examples, recommendations, tools, and a roadmap to help agencies advance their capability maturity in modern data management practices, which will support the development of TIM big data use cases and pipelines. Ultimately, agencies are encouraged to make a business case for TIM big data efforts like those presented in this research. To do so, they will need to consider questions like the following: • Does the big data approach bring enough value or benefit for the cost and efforts required? • How many staff and what resources will be needed for such efforts? • What if we are not ready to move to the cloud? What steps can be taken in the meantime? • If data are still limited, would embracing such data bring sufficient value to the agency?

Next: References and Bibliography »
Application of Big Data Approaches for Traffic Incident Management Get This Book
×
 Application of Big Data Approaches for Traffic Incident Management
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Big data is evolving and maturing rapidly, and much attention has been focused on the opportunities that big data may provide state departments of transportation (DOTs) in managing their transportation networks. Using big data could help state and local transportation officials achieve system reliability and safety goals, among others. However, challenges for DOTs include how to use the data and in what situations, such as how and when to access data, identify staff resources to prepare and maintain data, or integrate data into existing or new tools for analysis.

NCHRP Research Report 1071: Application of Big Data Approaches for Traffic Incident Management, from TRB's National Cooperative Highway Research Program, applies the guidelines presented in NCHRP Research Report 904: Leveraging Big Data to Improve Traffic Incident Management to validate the feasibility and value of the big data approach for Traffic Incident Management (TIM) among transportation and other responder agencies.

Supplemental to the report are Appendix A through Appendix P, which detail findings from traditional and big data sources for the TIM use cases; a PowerPoint presentation of the research results; and an Implementation Memo.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!