Managing State Transportation Research Programs (2019)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

84 This Synthesis study provides a broad view of how state DOT research programs across the country currently manage their programs for quality and value. No agency program is identical to another; each enjoys certain flexibilities and operates within specific constraints through its department mandates and unique research needs, limited budgets, organizational structures, external relationships, and staff composition. To place the programs on a common footing, this study developed a framework with four dimensions to analyze state transportation research programs: program capability, program management, program quality, and program value. By uncovering both “what” a research program is designed and structured to do as well as “how” the program accomplishes its functions, this study helps differentiate how research programs of different needs, structures, and processes deliver research quality and value over time. 4.1 Conclusions The researchers arrived at a number of conclusions corresponding to the four dimensions of the framework: Program Capability is the ability of the DOT to undertake research. This dimension touches on goals, knowledge, expertise, resources, organizational structure, and external stakeholder relationships. 1. DOT research budgets range widely from slightly less than $1 million to about $30 million per fiscal cycle. About half of DOT research program budgets are between $1 million and $6 million. Most research funding comes from State Planning and Research (SPR) Part B. The majority of research spending comprises the research areas of pavements and bridges or structures. 2. A research program’s role or size does not appear to depend on where it is situated within the DOT organizational structure. Most state transportation research programs tend to be located within the Planning or Chief Engineer Division of the state DOT, with a few exceptions located in more specialized divisions. Day-to-day working relationships between research program staff and other DOT employees dominate organizational reporting relationships. 3. Intentional planning for research continues to be an agency need. Research programs set strategic priorities and establish the overall direction of their research efforts in a variety of ways, but most wish to better engage with executive leadership and broader groups of research stakeholders to refine DOT needs and priorities. Some DOTs have research strategic plans to guide their program, but most research programs look to the broader DOT strategic plan and participate in conversations with DOT leadership and advisory committees. C H A P T E R 4 Conclusions and Further Research

Conclusions and Further Research 85 4. Research programs rely on external groups for research administrative functions. Agencies rely on national programs such as TRB—including the NCHRP—and AASHTO to conduct research and to provide training, publications, and surveys. In addition to these forums for knowledge sharing, ongoing research identification, and technical support, agen- cies periodically participate in peer exchanges to enhance their capabilities. The U.S. DOT’s FHWA provides additional resources and local office support to state transportation research programs, such as engagement in periodic research peer exchanges. 5. Research programs also use external groups frequently to perform the core research function. Universities conduct most research sponsored by DOT research programs across the country. University faculty may also participate on a research advisory committee and/or serve as a technical expert for projects. Non-university contractors conduct a small percent- age of research projects at most agencies. Only about half of research programs conduct any “in-house” research. 6. Desired research program staff skill sets include a fundamental research background or technical expertise, research administrative support competencies, and management and communication skills. Although technical expertise may vary depending on whether the research program conducts in-house research, a fundamental research background to ensure the technical quality of research with strong scientific rigor and inquiry is a core competency for research project managers. Research programs responsible for research support functions also require competencies in contracting, legal, accounting, and other administrative services. Additionally, management and communication expertise is needed to foster collaboration between different research stakeholders and to successfully admin- ister research contracts. 7. A DOT’s ability to innovate is closely tied to how other parts of the DOT champion and engage with the research program. Half of research programs consider “promoting agency innovation” as one of their main responsibilities. Research programs also manage knowledge of research methodologies, funding programs and resources, and library ser- vices to assist DOT staff with their needs. Very few agencies currently have programmatic responsibilities for staff training or continuing education to disseminate new knowledge, however. Program Management encompasses the processes and protocols that the DOT uses over the research lifecycle. This dimension covers different aspects of management, execution, and administration. 1. Research needs assessment processes continue to be a balancing act between top-down and bottom-up approaches. Most research programs currently utilize top-down processes through research advisory committees, and half of programs also engage in bottom-up open submission and voting processes. Many programs are supplementing these with stakeholder workshops and other exercises to refine research needs and to determine how best to channel resources. 2. Managing projects and administering the research program requires program staff to coordinate across many DOT divisions. Subject matter experts from various functional divisions often participate in research projects, as either idea champions, project managers, or on advisory panels. Ancillary research management and administrative functions such as finance, accounting, information technology (IT), and contracting and procurement tend to be conducted by DOT-wide groups outside the research program itself. 3. Research programs emphasize the importance of making the outcomes of research more accessible to stakeholders. Programs most often passively convey research outcomes through project final reports and program websites. Some DOTs develop research briefs, presenta- tions, and newsletters to not only engage the DOT’s own leaders but also to touch a wider audience, albeit in a limited manner.

86 Managing State Transportation Research Programs Program Quality is the rigor and diligence with which the DOT approaches research to make it efficient and effective. Following scientific principles, consistent research and experimental techniques, and other best practices fall under this dimension. 1. Most agencies focus on project-level measures of quality. This includes whether projects are completed in line with the budget and schedule, tracking project deliverables, formatting and presentation of findings in documents, and so on. 2. Quality control reviews provide additional rigor to research methodology and findings. Many programs rely on DOT project managers to critique the research deliverables, and some consistently establish technical advisory panels to provide oversight over technical aspects of research. 3. Agency research is typically not subject to peer review. This means that DOT research tends not to be reviewed by another researcher working in the same field of research. However, research studies published in journal articles could be peer reviewed but currently are not tracked. Some agencies are developing peer assessment programs to enhance research quality. 4. Some agencies are creating quality control standards. To standardize quality across the program, some agencies have created research manuals, payment schemes based on quality, and other methods. Others require robust quality control plans from researchers at the start of the project to introduce more specific measures of quality for the project. Program Value is the usefulness of new and enhanced knowledge and the degree to which it leads to an improved transportation system. This dimension focuses on research outcomes and impacts. 1. The barriers to external knowledge sharing and practitioner learning remain high. Research programs currently prioritize research dissemination within the DOT over the external dissemination of research findings. Programs tend to consider their state DOT— and in particular, executive leadership—to be the main audience of their research. External recipients such as local practitioners, other state DOTs, and universities or researchers are usually not dissemination targets. 2. Research programs often struggle to participate in research implementation. Programs have traditionally viewed their research “customers”—those who champion or request research on a topic—as responsible for translating research findings into practice. At the same time, programs often help manage or administer implementation activities by maintaining a list of completed research deemed implementable, coordinating the selection of imple- mentation projects, and administering related budgets. Not all agency research programs participate in implementation, and of those who do, only about half track project progress over time. Few agencies assess the long-term impacts of research implementation. 3. Research implementation by the DOT itself is most often unsuccessful due to the limited resources for implementation and DOT-wide barriers. The lack of (or very limited) fund- ing for implementation and lack of staff who are empowered to lead and held accountable for implementation are the main resource constraints. Wider organizational barriers include misalignment between research projects and DOT priorities, shifting priorities over time, or the inertia of the status quo. 4. Research and implementation performance are not linked to current or future research funding at most programs. Research evaluation is typically not used to direct or allocate research funding for future projects. Additionally, funding is not premised on successful implementation at most research programs. However, research evaluation is linked with individual researcher performance evaluations, and some research programs require quality control plans and consider implementability as a part of the project selection process. 5. Agencies are moving toward program-wide assessment of research outcomes and pro- gram value. More than half of agencies assess only project-level indicators and do not use

Conclusions and Further Research 87 program-wide measures. Of those that do report program-wide outcomes, more programs use quantitative than qualitative measures. Although quantitative measures may be easier to “roll-up” into aggregate program outcomes, agencies find that qualitative narratives can help document the effects of improved processes, lessons learned, the rationale for new standards and policies, and other lessons that may be difficult to quantify. State transportation research programs have various capabilities for undertaking research through their goals, knowledge, expertise, resources, organizational structure, and external rela- tionships. Most programs are interested in more effectively engaging with executive leadership and from the wider DOT to better understand agency needs and priorities. This allows more intentionality in setting a strategic research direction that is helpful and valuable to the depart- ment. Understanding agency priorities and needs also increases the likelihood of successful implementation when paired with appropriate resources through funding and supporting staff. Because of different program capabilities, different structures, processes, and protocols exist to execute, manage, and deliver the research function over the research lifecycle. Research pro- grams conscientiously organize themselves to consistently deliver research that is of high quality and value. Half of research programs also consider promoting agency innovation to be a key responsibility of their program, which implies a role for the research program to assist in the application of completed research. Ultimately, research programs that are able to communicate their program-level value and outcomes are able to gain influence among their departments and beyond as leaders in transportation research and innovation. 4.2 Future Research This study is intended to be a high-level analysis of state DOT research programs by synthe- sizing various topics that are normally researched on a standalone basis. The study team antici- pates that the information collected by this study will naturally generate new ideas for further research by the transportation research community. There are some possibilities for further research focused on the broader synthesis of research programs. Information provided in this study is based on information and perspectives from state DOT research program managers. A natural extension to this study could narrow focus on research visioning, program quality, and program value perspectives by surveying the individuals to whom research programs report to directly. The study team expects that much knowledge could also be gleaned by surveying individual research program staff and research project manager perspectives to better understand the nuances of program capability and management. Other areas for further research include additional case examples from other state DOTs through the same framework of program capability, management, quality, and value in order to further collective understanding on managing state transportation research programs.