SUMMARY

Evaluation of the Exploratory Advanced Research Program

This report presents the findings from an evaluation study conducted by the Transportation Review Board of the Exploratory Advanced Research (EAR) Program of FHWA. The EAR Program was launched in 2007 to fund research that could lead to breakthrough innovations benefiting the surface transportation system. Each year, the Program has funded multiple partnerships among universities, firms, and other organizations to investigate topics in disciplines spanning physical sciences, social sciences, and engineering. The study examined a sample of these research projects and their results to generate insights into the Program’s management and operational processes, its outcomes, and its effectiveness in driving positive impacts on the U.S. transportation research community and system.

Evaluation Approach

The evaluation team developed a database of 86 projects awarded by the EAR Program from approximately 2007 through 2019. The team then analyzed the EAR Program’s management processes over that time; the publications and other outputs generated by those projects; and the short-term and long-term outcomes of the research through data collected using a combination of methods:

• A review of documents related to the EAR Program produced by FHWA, research project teams, and TRB.
• A survey distributed to the principal investigator (PI) for each project awarded between 2008 and 2016; from 75 projects in that time frame, 64 PIs were contacted and 25 PIs provided complete survey responses.
• Interviews with program stakeholders, including FHWA Research and Technology (R&T) Program staff members, the FHWA project officers who provided oversight for EAR-funded projects, transportation research experts, and PIs for selected projects.
• Analysis of bibliographic records for scholarly journal articles, conference presentations, and patents generated from the results of EAR-funded projects.

The study team focused in particular on projects investigating three topics: cementitious materials, truck platooning, and video analytics. The team developed case studies encompassing the multiple projects funded for each of those topics and assessed the cumulative outcomes of research conducted in each area to illustrate the nature, scale, and scope of the EAR Program’s impact.

Findings

Overall, the EAR Program has made effective use of the funding allocated for extramural research. The Program has focused on topics perceived to be of high potential value to FHWA and the surface transportation community. As an exploratory program, the EAR

Program funds high-risk projects that often do not produce the expected outcomes. Still, EAR-funded research generated valuable outputs (especially in terms of new research tools and concepts) and catalyzed follow-on research investments into the topics selected. Relative to its scale of funding and staff, the EAR Program created a robust network of stakeholders and partners within FHWA and across federal and state governments, academia, industry, and the broader highway transportation community.

A potential issue complicating program management is the instability of the EAR Program’s budget from year to year. Based on funding availability, it is possible that well-qualified proposals submitted in a year with a lower budget would have been funded if submitted in a year with a higher budget. Budget instability also contributes to inconsistency across years in the proposal submission period and the ultimate award date for funded projects. These circumstances may discourage some proposers from submitting proposals to the EAR Program because they have no information about the likelihood of receiving an award and may not receive information about new Broad Agency Announcements in a timely manner.

The EAR Program produced specific intangible benefits, such as the development of new research software and datasets that benefit the transportation research community. The Program has also provided support to students and early-career researchers who then join the next generation of the surface transportation research workforce. The results of EAR-funded research contributed to policy and investment decisions on support for applied research and the development of emerging technologies at FHWA and elsewhere.

Uncertainties

The evaluation was hampered to an extent by incomplete and inconsistent data about the EAR Program’s past investments and their outputs. The study team undertook extensive efforts to reconstruct key facts, identify research results in technical publications and other documents, and trace later events to the contributions of EAR projects. The evidence collected may be incomplete due to lack of access to key people (who have retired or are otherwise unavailable to provide input), difficulty in obtaining survey responses from the project PIs, and the selection of topics for case study analysis.

A more significant factor is the nature of the research funded by the EAR Program. Because these projects are exploratory, they may not provide definitive answers to research questions and may produce unexpected results. The full significance and impact of exploratory research is often realized over many years and may be seen in developments that are only indirectly traceable to the EAR Program. This difficulty could be mitigated through careful and comprehensive record-keeping about research projects and their outputs, followed by retrospective investigations into the dissemination and use of their research findings.

Significance and Implications

The evaluation team developed a series of observations about the EAR Program that are intended to offer options to FHWA about the future of the Program:

• The EAR Program could benefit substantially from a more robust infrastructure for record-keeping and project tracking to enable management of the Program’s overall portfolio of research investments, aligning them with the strategic goals for the Program and FHWA. A portfolio management system would enable a view across multiple projects, budget years, and topic areas of how the EAR Program’s funding has been distributed

• across many areas of interest to FHWA. The system could also link projects to their key outputs (reports and other publications, patents, students trained, and research tools) and short-term outcomes, providing a more holistic view of the Program’s value than can be found in the current reports on Program results.
• With improved record-keeping, the EAR Program could do more targeted evaluation of its own processes and their results. It would be helpful to know, for example, whether better direction on transition support leads to more effective assistance to funded investigators and more productive transition activities. Program records could also include immediate research outputs (journal articles, conference presentations, patents, and students trained) associated with each project, offering more insight into the dissemination of knowledge generated by the EAR Program.
• Proposers, reviewers, and stakeholders could benefit from more clarity on how they can determine what research topics appear to qualify as exploratory advanced research. While uncertainty over the exact meaning of this term helps the EAR Program to maintain some flexibility and managerial discretion, it may also reduce confidence that the Program supports research that could not be funded through other channels. In addition, the EAR Program may be able to clarify its criteria for determining what constitutes exploratory advanced research by revising its taxonomies for clustering topics and projects and by undertaking retrospective analyses of how those criteria have changed over the course of the Program.
• A more fundamental issue is whether FHWA wishes to assign exploratory advanced research a higher priority in the FHWA R&T system. Doing so would require a commitment to providing more substantial and reliable funding and other resources, enabling the Program to gain greater visibility and leverage in the transportation research community. At a minimum, implementing some of the options for enhancement detailed in this report would require additional overhead funding and administrative support for the EAR Program staff. The Program runs efficiently with a lean staffing plan, so adding responsibilities may require additional personnel.