SECTION 4
Evaluation of EAR Program Effectiveness
The evaluation of the EAR Program’s effectiveness addressed the key question of whether the EAR Program is designed and conducted in a manner that is appropriate for supporting exploratory advanced research. As noted earlier, determining which research topics qualify as exploratory advanced research is not always a clear process. For this evaluation, the study examined the properties of the research topics funded by the EAR Program and assessed the extent to which the character of those topics aligns with the intent and purpose of the Program.
4.1 Evaluation of EAR Program Portfolio
The U.S. Department of Transportation Strategic Plan for FY 2018–2022 (U.S. DOT, 2018) established four overall goals for U.S. DOT and, by extension, for FHWA:
- Safety. Reduce transportation-related fatalities and serious injuries across the transportation system.
- Infrastructure. Invest in infrastructure to ensure mobility and accessibility and to stimulate economic growth, productivity, and competitiveness for American workers and businesses.
- Innovation. Lead in the development and deployment of innovative practices and technologies that improve the safety and performance of the nation’s transportation system.
- Accountability. Serve the nation with reduced regulatory burden and greater efficiency, effectiveness, and accountability.
In conjunction with its legislative mandate, the EAR Program is directed to support “longer-term, higher-risk research with potentially dramatic breakthroughs” related to these four goals. The Program addresses the Innovation goal explicitly. By funding projects oriented toward technological breakthroughs, the EAR Program could (if successful) position FHWA and U.S. DOT to lead in the development of innovative practices and technologies.
The goals of Safety and Infrastructure provide direction on specific priority areas of research for the EAR Program. These two goals also align with two of the FHWA R&T Offices (the Office of Safety and the Office of Infrastructure). Reviewing the entire set of topic areas published in past EAR Program BAAs, the following focused squarely on Safety and Infrastructure:
- Development of a New Data Analysis Method in Support of Integrated Safety System (ISS) for Highway Safety,
- Development of Methodologies to Evaluate the Nighttime Safety Implications of the Roadway Visual Scene Under Varying Cognitive Task Loads,
- Greatly Increased Use of Fly Ash in Hydraulic Cement Concrete (HCC) for Pavement Layers and Transportation Structures,
- Nanoscale Measurement Devices for Advancing Understanding of Highway Pavements,
- Development of Enhanced Safety Systems Based on GPS/INU System,
- Innovative and Environmentally Beneficial Infrastructure Materials,
- Modeling Cement Hydration Kinetics,
- Virtual Nondestructive Evaluation (NDE) Laboratory for Highway Structures,
- Testing Methods to Reduce Distraction from Hand-Held or Mobile Devices, and
- Supplementary or Alternative Materials for Highway Pavements and Structures.
Other topics were clearly designed to support enabling technologies that contribute to research aimed at these goals. For example, the projects on VA were intended to create tools for exploiting rich visual data in safety research. Projects funded under the Mobile Ad Hoc Networks (MANETs) topic explored the use of flexible wireless communication systems for monitoring and safety applications. Many of the projects in topics related to connected vehicles have important implications for safety.
Addressing the goal of Accountability has two elements relevant to the EAR Program. First, some research topics focus on improving the overall efficiency of the national highway system through better intelligence, coordination, and forecasting. A number of topics related to traffic modeling, systems-level simulation and analysis, and data analytics support that capability. Research in areas such as behavioral economics and new approaches to toll collection can inform policy decisions, ensuring that regulations are designed to be less intrusive and more effective in overseeing highway traffic and driver behaviors.
Other topics have tenuous connection to these goals but still represent potentially significant breakthroughs. Research on energy topics, such as technologies and policies for energy conservation, can lead to important benefits to overall U.S. transportation infrastructure and broader social goals related to environmental impact.
The EAR Program has categorized its selected research topics into a set of clusters. This taxonomy is somewhat contradictory because it includes research fields or disciplines (materials science, human behavior), technologies (connected highway), and applications (energy and resource efficiency, assessing performance). The EAR Program could develop a revised taxonomy aligned with FHWA strategic goals, providing clear justification for topic selection. That taxonomy could also guide potential proposers on how they could connect their envisioned projects to agency priorities and needs.
Within those topics, the EAR Program could select individual projects that show the potential for breakthrough improvements related to each goal. The scoring rubric for proposal evaluation contains clear instructions on how reviewers should judge the potential for achieving both breakthrough innovations and gains to FHWA priority areas. Both of these measures seem to be effective in focusing reviewers on selecting projects that fit the criteria for exploratory advanced research.
4.2 Effectiveness of EAR Program Support for Research Transition and Impact
A second dimension of effectiveness concerns the capacity of the EAR Program to facilitate the transition of research into downstream development efforts and then to produce impacts. As noted in the process evaluation, the EAR Program has improved its processes and frameworks that guide transition activities and screen project results for transition potential. The Program has a limited budget, so it needs to be selective in choosing which projects receive enhanced transition support. Since these processes have been changing in recent years, evidence is not sufficient to reach clear conclusions about the Program’s effectiveness in this area. Results from the PI survey suggest that the perceived effectiveness has been mixed. The EAR Program could
undertake a more systematic retrospective review of what projects are selected for transition support and the results of those efforts in advancing technology development.
To improve accuracy, such a review could be sensitive to situations where transition is impeded by factors beyond the control of the Program. For example, in TP, experts and PIs noted that the rapid progress in developing fully autonomous vehicles drew attention away from partial automation and reduced the interest in this technology in the private sector. A more comprehensive evaluation of transition effectiveness would analyze not only transition plans but also actual transition support activities undertaken by EAR Program staff and PIs.
One measure that could be considered is more direct training on transition for PIs themselves. Some comparable research programs at other federal agencies provide this type of support. At the NSF, the Innovation Corps (or I-Corps) Program puts PIs through a “boot camp” process on preparing their research results for commercialization. The Program features a strong emphasis on conversations with potential end users. Similarly, the Homeland Security Advanced Research Projects Agency at the Department of Homeland Security trains PIs on how to formulate the “value proposition” for their projects. The training includes a structured approach to creating an “elevator pitch” for each project so that a PI can state concisely and clearly the potential and expected benefits of the project and make a case for why it is superior to other comparable research.
4.3 Assessment of EAR Program Impact Relative to Objectives
The EAR Program’s overall outcomes and impact are difficult to encapsulate given the very broad range of research supported and the relatively low amount of funding available. Other federal research programs with similar objectives have much larger budgets. For example, DARPA invests approximately $8 billion per year across a few dozen research programs focused on breakthrough innovation. With that scale of investment, those programs can attract the critical mass of researchers and organizations needed to catalyze the emergence of new research communities focused on completely novel fields and concepts. While individual EAR grants are larger than a standard NSF grant, the number of awards issued and the total funding invested per topic area by the EAR Program are not sufficient to attract a broad spectrum of leading researchers.
Relative to its scale, the EAR Program appears to be somewhat effective at supporting high-impact research. Because its projects have modest resources, the Program seeks primarily to test the feasibility of potential breakthrough concepts rather than ensure that they come to fruition. This screening function has clear value to FHWA. Investments in research that appears promising but is ultimately infeasible would waste resources and time and could result in duplicative efforts. The challenge for the EAR Program is in determining which research opportunities have real potential for breakthrough results. This process is conducted primarily through qualitative assessment by proposal reviewers, as well as by assessments of topics through ISIs. Given the number of projects funded so far, the EAR Program has accumulated data and experience that could be used to develop more informed guidelines on making those judgments.
4.4 Findings from Program Effectiveness Evaluation and Potential Options
In summary, the EAR Program appears to have improved its management and processes over time, with apparent gains in its overall effectiveness and potential to produce substantive impact from surface transportation research. The EAR Program is seen as filling a unique role
in the research community by focusing on topics that are not purely fundamental academic research but are too speculative to be appropriate for applied research funding. In the short term, FHWA has several options for implementing better monitoring and assessment efforts to improve Program processes. A more fundamental issue is whether FHWA wishes to assign exploratory advanced research a higher priority in the FHWA R&T system. Doing so may require a commitment to providing more substantial and reliable funding and other resources, enabling the Program to gain greater visibility and leverage in the transportation research community.
