APPENDIX C

Case Study on Video Analytics

C.1 Definition and Timeline

Video analytics (VA) tools are used across several domains, such as national defense, law enforcement, and robotics. For years, computer vision researchers have been investigating transportation-related issues and roadway/driver videos and models for issues such as driver distraction. Federal agencies, universities, industries, and national labs were engaged in funding and conducting initial VA research.

TRB developed the initial Strategic Highway Research Program (SHRP). A second round of the program, known as SHRP 2, addressed improving highway safety, reducing congestion, and improving methods for renewing roads and bridges. Among other activities, this program funded the Naturalistic Driving Study (NDS), where cars were equipped with video cameras and other monitoring technology to observe human behavior under different driving conditions. Since the video captured by the NDS includes images of drivers and other personally identifiable information, the data must be stored securely. To that end, in 2015 the Safety Training and Analysis Center was created to house the SHRP 2 dataset; the Center is located at the TFHRC and is supported by the Virginia Tech Transportation Institute. FHWA began to take a stronger leadership role in managing and leveraging SHRP 2 data.

FHWA, through the EAR Program, sought to leverage existing analytics tools of computer vision and VA to help improve the annotation, use, and accessibility of the massive volume of data captured by the NDS. Automated feature extraction software becomes more relevant as it becomes more challenging to manually check visual data. The EAR Program came into the picture because it supports crosscutting programs and can fund projects that do not have well-defined end products.

C.2 VA Research: Initiation and Evolution

In 2011, the EAR BAA included VA as a topic. Although SHRP 2 data were not yet available, projects anticipated that broader use of that data would require processing NDS data and other large repositories of video footage. Research themes included identifying vehicles, detecting eye movement, and analyzing road conditions, as well as analyzing the increasing abundance of visual data from highway and roadside cameras. Panelists for technical proposal evaluation come from within FHWA (Office of Safety R&D).

C.2.1 Rationale for VA as an EAR Topic

The EAR Program realized that the limitations on access to the SHRP 2 data due to privacy and confidentiality issues would hamper research using video footage. To allow for more widespread

distribution of datasets, the EAR Program would need two key capabilities: the ability to de-identify or mask drivers’ facial features and vehicle identity and the ability to extract key variables (features) embedded in video footage for later analysis. By supporting research in VA, the EAR Program created a valuable new data infrastructure for safety researchers that could generate substantial follow-on research.

C.2.2 VA Project Selection

In 2011, the EAR Program issued the first BAA that included VA as a specified topic. Although data from the NDS were not yet available, this topic was selected to anticipate the need for processing large volumes of video data. Research projects were solicited that could build on the breakthrough techniques in software, funded by FHWA and others, related to identifying vehicles, detecting eye movement, and analyzing road conditions, as well as analyzing the increasing abundance of visual data from highway and roadside cameras. Panelists for technical proposal evaluation came from within FHWA (Office of Safety R&D), other government agencies [Oak Ridge National Laboratory (ORNL), NIST, and Volpe National Transportation Systems Center], and industry (Toyota). The first VA project was awarded to Carnegie Mellon University in 2012 for “Machine Learning Analysis of Large Volumes of Highway Video.”

By 2013, the scope for VA projects in transportation research gained clarity. In addition to automating extraction of features, researchers recognized the need to de-identify data to make the datasets available without restrictions on data access. Features related to safety were emphasized, specifically driver awareness factors such as head pose, eye gaze and movement, hand detection, seat belt detection, and cell phone detection. Three projects were funded to focus on feature extraction, and two projects were funded for identity masking.

The 2018 BAA was a further continuation of 2013 themes of feature extraction for in-cabin and external video. Three projects were funded for that round related to VA. Two more projects, while not falling under the VA topic of the BAA, are included because they relate to the influence of results of VA projects on the realistic models that are the basis of artificial realistic datasets. There were six initial projects and five that followed on after those, given continuing interest and the promise of this area of research. Details of these projects are provided in Table C-1.

C.2.3 VA Project Oversight and Management

The EAR Program has made sustained investments in VA projects, which includes substantial project support. The Program provided funding to ORNL to conduct IT assessments of VA software, consult on transition planning, and conduct quality and performance tests on the software. Oak Ridge researchers noted that the EAR-funded VA projects were funded at the beginning of research on artificial intelligence, such as deep learning, which has revolutionized video data analysis in general, thus limiting the appeal of those packages.

C.3 VA Research Outcomes

The major outcome from VA research at the EAR Program is a new set of research tools for the academic community. Research outputs include progress in the following functionality of tools:

• Identify, classify, code; algorithms automate processing
• Features covered: driver/other; mask driver’s identity
• Rate of specific factor/state in video; identify factors of specific incident

Table C-1. VA-related projects funded by the EAR Program.

• Test, assess effectiveness of tool
• Prototype systems; test application to other features
• Provide segments of video data

EAR’s funding has had a significant impact on developments in VA related to transportation research. One of EAR’s most significant contributions has been connecting researchers with other collaborators to expand research in this field. EAR’s funding supported dozens of early career researchers and helped introduce them to transportation research.

SHRP 2 video data are somewhat crude, but the data are still comprehensive and useful. The data continue to be useful despite concerns that the data would become obsolete by this point. Projects from the second round of BAAs leveraged findings from the first round to automate datasets. Now there is a lateral shift in focus from SHRP 2 data to using findings to construct and validate realistic artificial datasets and to make these more accessible. According to technology evaluation experts at ORNL, the second round has gotten a lot closer to processing the entire SHRP 2 dataset than the first; there are now more tools in use.

The role of EAR as a conduit for collaboration in the VA space is very important. Two survey respondents reported forming new partnerships as a result of EAR project funding, and one reported a new partnership. One PI reported that their work was leveraged into further work with DARPA and industry partners. Another PI reported that EAR allowed them to build collaborations and expand expertise in computer vision for transportation problems. Several researchers reported that their research could be of interest to multiple stakeholders. New research methods

developed by these EAR-funded projects could be of use to FHWA, state DOTs, other federal agencies, universities, and industry.

Most respondents reported that EAR funding supported early-career researchers at the undergraduate, graduate, and postdoctoral levels. Three projects supported postdoctoral research; each project supported one to two researchers. Five projects supported graduate-level research, ranging from one researcher to five graduate researchers. Interestingly, four projects supported researchers as early as the undergraduate level, with two projects supporting more than 15 researchers. In comparison with other EAR-funded projects, VA projects supported, on average, six times more undergraduate students and two times more graduate students. This is probably because the research in this area involves less access to costly equipment and lab facilities, which are required for areas such as infrastructure.

To date, the VA projects have produced 27 related publications and conference presentations. The projects also led to two patent inventions granted to SRI International, including one for a system to monitor driver awareness.