SECTION 1

Introduction

At the request of the Research and Technology (R&T) Program of FHWA, TRB conducted an evaluation of the Exploratory Advanced Research (EAR) Program. The kickoff meeting for the project was held at the Turner-Fairbank Highway Research Center (TFHRC) in February 2020, and the evaluation team concluded its data collection and analysis in October 2021. The evaluation was led by RTI International, a nonprofit research organization.

This report provides a summary of the evaluation project, including an overview of the project activities and findings. The report is organized as follows:

• This section provides a review of the establishment, development, and design of the EAR Program since its launch in 2007. It also presents the design of this evaluation and the methods employed.
• Section 2 reports on the evaluation of the EAR Program’s processes, including a review of the Program’s activities and investments and an assessment of the Program’s strategies and management.
• Section 3 examines the outcomes and impact attributable to the EAR Program’s research funding and coordination activities, focusing on the period from 2008 through 2019. This section includes findings from the more detailed analyses of EAR Program investments in three research topics [cementitious materials (CM), truck platooning (TP), and video analytics (VA)] and a cross-case analysis.
• Section 4 provides an assessment of the effectiveness of the EAR Program as of 2021 in addressing its intended objectives and its contribution to fulfilling FHWA’s strategic goals. It also discusses options for FHWA to consider as the agency formulates plans for enhancing and improving its support of exploratory advanced research in the surface transportation domain.
• Significant components of the data collection and analysis for this evaluation are included in the report appendices. The appendices include case studies on the three topics examined in detail and the survey distributed to principal investigators (PIs) of prior EAR projects.

1.1 Background Information on the EAR Program

Within FHWA, the R&T Program supports scientific research and technology development by addressing key knowledge gaps and accelerating technological innovations to meet current and future needs in the U.S. highway transportation sector (FHWA, 2018a). Beginning in the 1980s, the Research and Technology Coordinating Committee (RTCC) of TRB issued a series of reports that called attention to the relatively low level of funding at FHWA for what the committee termed “breakthrough research.” While R&T Offices focused substantial efforts on relatively near-term technologies and concepts—those expected to be ready for implementation in

the surface transportation sector within a few years—comparatively little investment had been made in more high-risk, speculative development efforts. This breakthrough research would take longer to reach maturity but would also leverage revolutionary methods, ideas, discoveries, and inventions to generate disproportionately greater benefits to the transportation sector and to the nation. The RTCC noted that the funding for breakthrough research that did exist was concentrated in the Intelligent Transportation Systems (ITS) Joint Program Office (JPO) and, thus, directed at only a portion of FHWA’s objectives.

SAFETEA-LU established the EAR Program to address that gap. The Act directed U.S. DOT to establish the EAR Program to address “longer-term, higher-risk research with potentially dramatic breakthroughs for improving the durability, efficiency, environmental impact, productivity, and safety (including bicycle and pedestrian safety) aspects of highway and intermodal transportation systems” (U.S. Congress, 2005). The Program was authorized at $14 million per year for the years 2005 through 2009 and subsequently reauthorized by the Moving Ahead for Progress in the 21st Century Act in 2012 and the FAST Act in 2015. The authorizing language for the EAR Program specified that it should promote collaboration between FHWA and other public- and private-sector organizations. In keeping with those legislative requirements, the EAR Program’s main activity is to fund extramural research (research conducted at organizations other than FHWA) and use that funding to engage universities, private firms, independent research institutes, and nonfederal government agencies in longer-term research.

The EAR Program issued its first annual call for proposals from research organizations in 2007, using a research solicitation mechanism called a Broad Agency Announcement (BAA). Developed for funding programs at the Defense Advanced Research Projects Agency (DARPA), the BAA solicitation requests initial “concept papers” from proposers that provide the broad parameters of the projects they plan to conduct, including the expected results and their potential benefits. Each concept paper is reviewed by FHWA experts familiar with the proposed topic, and authors of promising concepts are invited to submit a full proposal. Those proposals are evaluated by a panel of experts from FHWA and other organizations based on three criteria: scientific or technical merit, importance to FHWA programs, and alignment of the project’s budget with available EAR funding.

The 2007 BAA requested proposals across a broad range of topics relevant to FHWA’s mission, including human-automation interaction, safety improvements through advanced data analysis, methods to improve transportation system resilience, and technologies for energy efficiency and development of alternative fuels. This call elicited proposals covering a broad range of topics. Beginning in 2008, the EAR Program adopted a practice of selecting three to five research concepts or topics to highlight in each BAA, providing each topic’s background, parameters, and significance. While proposers were still welcome to address any topic, almost all proposals were aimed at one of these specific topics. Topics were selected based on their potential to generate a breakthrough discovery or technology with significant benefits to areas relevant to FHWA’s mission. The EAR Program developed a semi-structured process for topic selection involving a range of EAR Program stakeholders, as described in Section 2.

Since 2008, the EAR Program’s BAAs have featured topics covering a broad range of technical domains and disciplines. A sample of topics published in EAR Program BAAs is provided in Table 1-1. While the Program’s enabling legislation authorized spending of up to $14 million per year, the Program’s annual budget varied between $5 million and $14 million; thus, it could fund only 6 to 10 projects per year. As a result, the Program frequently had more projects that merited funding, based on reviewers’ scores, than it could support. This also limited the amount of funding invested in each topic area to $2 million to $3 million. The relatively modest funding amount fits the Program’s mission to explore a topic’s potential rather than to fund the development and implementation of new technologies. In a limited number of cases, the Program conducted a

Table 1-1. Selected topics listed in EAR Program BAAs.

follow-on solicitation for a particular topic when the initial funded projects generated evidence of high potential. In most cases, the Program funded a topic and then moved on. To leverage its funding, the EAR Program required some proposers to meet a cost-sharing requirement, where the performing team’s partners would provide their own support through direct funding or in-kind contributions (such as complimentary access to data, equipment, or personnel). Cost-sharing contributions by industry partners in particular were encouraged, since those investments were viewed as signals of a project’s commercial viability (with a company’s funding as evidence that a firm saw market potential for that project).

The EAR Program also implemented a notable requirement for proposals. Each project must include a “transition plan” outlining steps that the investigator(s) will take to identify and contact organizations that could fund or conduct follow-on research if the project shows promising indications of breakthrough potential. These organizations are termed “transition partners.” To assist such efforts, the EAR Program would fund independent assessments of a concept’s Technology Readiness Level (TRL) or, for software, an assessment of the program’s robustness. These assessments were conducted by independent groups funded by the Program at institutions such as DOE’s Oak Ridge National Laboratory (ORNL). The EAR Program also created plans for communicating project results or facilitating the handoff of research to other funding sources. Projects are publicized through official FHWA publications, such as specialized brochures (TechBriefs) or features written for the FHWA R&T magazine, Public Roads.

During the funding period for an EAR-funded project, an Agreement Officer’s Technical Representative (AOTR) provides oversight of the team’s progress, often through telephone calls and periodic written reports. These individuals are often research staff in TFHRC Offices who provide oversight in addition to their regular duties. They also help disseminate EAR project findings within FHWA and to others in the surface transportation domain (such as state and

local transportation authorities), and they facilitate contacts with potential transition partners. More details on the duties of AOTRs can be found in Section 2.

As EAR-funded projects focus on investigations of concepts at the earliest stages of development, they rarely produce tangible products or solutions. The evaluation team found that project results are encapsulated most frequently in articles published in research journals related to transportation or in presentations at research conferences. At the end of the project, the research team must submit a final project report. These reports are often available through the U.S. DOT’s online document repository.

Since 2008, the EAR Program has been led by its current Program Director, David Kuehn. Kuehn has overseen the selection of topics published in the BAAs, the convening of review panels to evaluate proposals, negotiations with selected investigators over the terms of their funding agreements, publicity and communications for the Program, and coordination with other funding agencies and outside contractors. Kuehn also oversees administration of the National Research Council Research Associateship Programs (NRC RAP) at FHWA, which brings postdoctoral researchers to TFHRC for limited-term appointments to conduct discrete research projects. The NRC RAP is not directly related to the EAR Program, but there are some synergies that result from connecting the two.

Total investment in the projects from fiscal year (FY) 2007–2020 amounts to over $65 million. After nearly 15 years in operation, FHWA decided that this would be an opportune time to assess the aggregate impact and benefits generated by the EAR Program and consider potential options for the future.

1.2 Evaluation Purpose and Design

The primary purpose of this evaluation was to provide FHWA with an assessment of the impact of the EAR Program from its inception to the present and to generate findings that will inform potential future decisions about the strategy, structure, and management of the EAR Program. To provide such insight, the project combined evaluations of three related aspects of the EAR Program:

1. Program processes. How has the EAR Program’s structure and operation supported exploratory advanced research, and are its processes appropriate for that type of research?
2. Outcomes. What are potential and realized outcomes of EAR Program–funded research and any follow-on research and development (R&D) efforts generated from that research?
3. Program effectiveness and future options. Have the EAR Program strategies and implementation supported achievement of its objectives? What might be done differently in the future?

This evaluation differed from other evaluation activities in the FHWA R&T Program, such as recent reports produced by the Volpe National Transportation Systems Center, in that it encompassed all the activities of the EAR Program, rather than focusing on its investments in particular technologies or research areas. As a consequence, this project analyzed the aggregate impact and benefits generated across all research projects funded by the EAR Program, regardless of discipline or domain.

Because the EAR Program funds research that is often speculative and risky (in technical terms), the evaluation did not judge the “success” of individual projects, nor did it assess whether the Program should have selected proposed projects other than those awarded. Instead, this evaluation assessed the degree to which the Program’s portfolio aligns with the concept of “exploratory advanced research” and whether the benefits generated by the results from that portfolio are reasonable given the nature of the research.

1.3 Overview of Evaluation Methodology and Activities

The EAR Program funds research projects covering a broad variety of research topics, methods, and objectives. These projects produce intangible outputs—in the form of new knowledge, methods, theories, and frameworks—that are difficult to measure in quantitative terms. R&D projects are conducted in a complex environment, where the outcomes can be affected by other concurrent research, the actions of many different stakeholders, and uncertainties about their potential findings and application. Following the guiding document produced by the Research, Technology and Development Evaluation Topical Interest Group of the American Evaluation Association, this evaluation employed a “mixed-methods” approach to data collection and analysis (Research, Technology and Development Evaluation Topical Interest Group, 2015). As shown in Table 1-2, the data sources for this study included a range of sources applicable to different elements of the evaluation. The main sources of data included:

• Program documentation, including EAR Program BAAs, summaries of the proposal panel reviews for specific Program years and topics, EAR Program Factsheets, internal Program documents [including TRL assessments, information technology (IT) assessments, Program communication and dissemination plans, project funding agreements, and project final reports].
• A survey distributed to the PI for each project awarded between 2008 and 2016. From 75 projects in that time frame, 64 PIs were contacted and 25 PIs provided complete survey responses.
• Interviews with Program stakeholders, including FHWA R&T staff members, the FHWA project officers who provided oversight for EAR-funded projects (AOTRs), transportation research experts, and PIs for selected projects.
• Analysis of bibliographic records for scholarly journal articles, conference presentations, and patents generated from the results of EAR-funded projects.

1.3.1 Notes on Document Sources

Program documents were provided by the EAR Program Director and other Program staff and obtained from the FHWA R&T website and the National Transportation Library’s online document repository. The evaluation team constructed an inventory of funded projects from

Table 1-2. EAR Program high-level evaluation matrix.

summary spreadsheets provided by the EAR Program and then matched the projects with records in the FHWA R&T online database and other sources of FHWA spending records. The project records provided were incomplete in some cases, and in several instances, they contained contradictory information (such as naming the incorrect PI or identifying the wrong AOTR). Records from the FY 2007 funding competition were omitted from the Program’s records. Some discrepancies resulted from changes during the course of a specific project—for example, if a project’s proposed budget differed from the final awarded budget after negotiations and then could be further revised through add-on awards and extensions.

To facilitate the evaluation data analysis, all project data were summarized in a set of spreadsheets that assigned a unique identifier to each project and linked the project to significant data (such as PI names, performing organizations, project title, funding duration, and awarded amount of funding). The evaluation team added supplementary fields, such as a classification of the research area for each project, and links to each project’s documents. This database formed the basis for the analysis of proposals and funded projects provided in Section 2.

1.3.2 Case Studies as an Evaluation Tool

As a means of integrating these disparate data sources into a cohesive evaluation, the team compiled case studies of three technical areas where the EAR Program made sustained investments. The case study selection process placed an emphasis on illustrating a diverse set of research areas, with differing technological implications, project objectives, and Program results. These case studies are not intended to be representative of “typical” EAR funding efforts. Instead, they show how the EAR Program took differing approaches to different technical domains and how those efforts led to a combination of purposeful and serendipitous impacts.

Case studies are useful for conducting evaluations because they can delve into nuances and conditions that are not apparent when analyzing summary data or Program documents. This study used case studies to explore the processes through which the EAR Program made its investments, the relationships between key stakeholders in each set of projects, and the dynamics that affected the results obtained and the benefits generated and disseminated. The research areas selected for the case studies are

• Cementitious materials (CM)—a series of projects that investigated methods for developing and evaluating materials that could supplement or replace conventional Portland cement to provide new properties (such as greater durability).
• Truck platooning (TP)—a set of projects focused on the requirements and parameters for real-world deployment of systems that allow freight trucks to travel in a convoy with very short intervals between trucks, with automated controls that prevent those trucks from colliding during maneuvers.
• Video analytics (VA)—the application of software for machine vision and extraction of features from video footage that can take very rich video data and reduce it to a format that facilitates the analysis of driver behavior, road conditions, and other elements important to vehicle safety.