Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

III-E-1 Implementation Plan A P P E N D I X E The guidance in this document for the continued assessment and improvement of traffic forecasts will only be helpful if it is implemented. The following pages describe potential barriers to implementation, offer ways past those barriers, and proposes a plan to promote the implementation of this research. Editor’s Note: In Appendix E, “we” refers to the project team that conducted the research for NCHRP Project 08-110 and authored NCHRP Research Report 934. 1 Lessons from Data Sharing Research The key part of the successful implementation of this research is the willingness of transportation agencies to share data about their forecasts, as well as the actual outcomes of their projects. In understanding this challenge, agencies and forecasters can draw from the lessons of other fields to understand the factors that affect the decision of whether or not to share data, and adapt those lessons to promote data sharing in traffic forecasting. Consider the example of scientists’ willingness to share their research data. These are datasets that would be archived electronically, in addition to published papers, and that would allow other scientists to more easily reproduce, review, and build upon their work. From a community perspective, data sharing is a clear winner, but some individual scientists may be reluctant to share either because of the additional effort involved, because it opens up the possibility of someone else finding an error they made, or because they may perceive exclusive access to certain data as a competitive advantage. There are similarities to our problem where there are clear community-level benefits to data sharing but individual agencies may be reluctant to share, either due to the effort involved or because they may not want to be criticized using their own data. Figure III-E-1 shows the results of recent research quantifying the factors contributing to scientists’ data sharing behavior (Kim and Stanton 2016). The boxes indicate the factors identified, and the numbers indicate the relative importance of each factor, with larger numbers being more important. Several observations can be made from these results. The two largest factors are scholarly altruism and normative pressure (the norms of whether others in their discipline share data). Both show up as having a larger influence than regulative pressures. In addition, the perceived effort associated with data sharing is an important impediment, whereas the availability of a data repository has a positive (but insignificant) influence.

III-E-2 Traffic Forecasting Accuracy Assessment Research Figure III-E-1. Factors affecting data sharing behavior (Fig. 2 in Kim and Stanton 2016). Important differences exist between the data sharing behavior of individual research scientists versus that of transportation agencies—particularly with respect to the perceived risk of criticism. In part, this is because science has a longstanding culture of critical debate—being transparent and open to argument builds the credibility of the science itself. Promoting a culture of constructive engagement that aims to improve and inform rather than to blame is essential to reducing agencies’ reluctance to share data. Research into the factors influencing the adoption of new travel forecasting methods (Donnelly et al. 2010) offers similar lessons. In particular, one of the biggest factors influencing agencies’ decisions to adopt new travel forecasting techniques was the presence of an individual who would serve as a champion. The lesson here is to recruit individuals to champion the implementation of this research. Taking these lessons together, the project team suggests that the key elements necessary to promote the implementation of this research are as follows: Develop a working data repository, as described in Part I, Chapter 3 of this report. Make it widely available and make it easy to use. This serves the dual goals of having a data repository and of reducing the perceived effort associated with data sharing. The authors hope that the Forecast Cards and Forecast Card Data represent the beginning of a growing and sustainable resource. Identify and recruit individuals to serve as potential champions for the implementation of this research at their own agencies. Make an appeal to these individuals focused on the greater good of the field.

Appendix E: Implementation Plan III-E-3 Minimize the perceived risks of sharing data by promoting a culture of constructive improvement. This can be achieved, in part, by having the early adopters talk publicly about the benefits they see from participating. Once the early adopters are participating, take advantage of their participation as an emerging norm in the field that will further incentivize others to participate. With a strong focus on this approach, regulative pressure (as shown in Figure III-E-1) may not be necessary. 2 Impediments to Successful Implementation Several specific barriers may impede the successful implementation of this research, but paths also exist to move beyond those barriers. Barrier 1: Lack of Awareness Potential users of the Forecast Cards and Forecast Cards Data will not implement this research if they are not aware of it. Therefore, a strong dissemination push is warranted. Ongoing efforts might group meetings, and publishing academic papers derived from this research. Barrier 2: Lack of Resources We must be honest in acknowledging that effort is involved in implementing the recommendations in this guidance document. The benefits of doing so include the potential for a more efficient allocation of resources toward more effective modeling techniques, but the agencies responsible for forecasts must still invest the resources to archive the information and analyze the accuracy of their forecasts. Naturally, the less cumbersome it is to implement the recommendations, the more likely agencies are to do so. Accordingly, the project team strove to find ways to make implementation as easy as possible, from developing the archive and information system and providing annotated outlines for the Silver-level archiving standard to conducting deep dives. The resulting steps and tools were tested during the NCHRP 08-110 workshop, but would benefit from more extensive beta testing and associated refinements. Barrier 3: Fear of Transparency One potential barrier to implementation is that agencies responsible for traffic forecasting may fear that by examining forecast accuracy, they are opening themselves up to criticism if those forecasts are shown to be inaccurate. Part I, Chapter 1 of this report discusses this issue, as well as the potential for a transparent examination of forecast accuracy to be used as a tool for building credibility. There is an aspect to this issue that relates to culture and norms. As noted in Figure III-E-1, normative pressure is the biggest influence on scientists’ willingness to share research data. Normative pressure means that people are more willing to take an action if they see their peers doing so and perceive it to be the norm. It is reasonable to expect similar dynamics are at play in traffic forecasting, with a similar effect observed when examining the emergence of activity-based travel demand models (a type of traffic forecasting model that is more methodologically sophisticated than traditional trip-based travel demand models). The first few activity-based models came slowly, with many potential adopters not wanting to take a risk on a new method. Once a handful of agencies had adopted involve presenting the findings at national and international conferences and at local model users’

III-E-4 Traffic Forecasting Accuracy Assessment Research such models, they talked publicly about their experience and the benefits they saw in doing so. Other agencies felt safe to join and could build upon the development efforts of the early adopters. This led to the implementation of such models accelerating, particularly among larger metropolitan planning organizations. Considering this history, one of the best things that can be done to promote broader implementation of this research is to do everything possible to get a handful of early adopters to implement the recommendations and see success in doing so. Those early adopters will then serve as examples for others, both in terms of further testing and refining the process and in terms of providing safety in numbers. 3 Proposed Activities The following activities are proposed to promote the implementation of this research. Task 1, Dissemination, had already begun during the closing months of this research project. The other tasks will require additional resources. Task 1: Dissemination The dissemination activities aim to increase awareness of this research. They focus on traditional scholarly dissemination strategies, such as conference participation and dissemination through journal articles. Specific activities include: The research was the focus of a Sunday workshop on “Progress in Improving Travel Forecasting Accuracy” at the 98th Annual Meeting of the Transportation Research Board in Washington, D.C., in January 2019. This research will be the focus of the closing plenary session at the TRB Planning Applications Conference in Portland, Oregon, in June 2019. This research will be presented at the Florida Model Task Force Meeting in July 2019. The project team has drafted a journal article referencing NCHRP Research Report 934 and describing the quantile regression analysis. This journal article will be submitted after approval of the final report and in coordination with NCHRP. The project team plans to publish additional journal articles referencing this NCHRP report, in which the authors will identify additional opportunities to present the work. Task 2: Implementation Site Visits Achieving the first few successful implementations of this research at transportation agencies is critical. One way to achieve these early successes is to identify potential early adopters and provide support to establish the process at those agencies. As currently envisioned, implementation support could involve a 2-day site visit by 1–2 members of an implementation team. During this site visit, the implementation team could work directly with agency staff to teach them how to do the following tasks: Calculate the uncertainty ranges around forecasts, Install the forecast card repository so they can track their own projects,

Appendix E: Implementation Plan III-E-5 Enter the first few projects into the repository according to the Bronze-level archiving standard, Work through the documentation needs of the Silver-level archiving standard, and Work through an example by analyzing the data using the data from this report. The site visit also could include a briefing for management-level staff to discuss the goals and expectations of the effort. Following the site visit, the implementation team would be available for telephone/web-conference support in the event that additional issues arise. Task 3: Beta Testing Refinements As with any project that involves software and new processes, early users may identify refinements that would make the process more usable for their own needs. This is a normal and important part of the development process. For this reason, the recommendations have been written with the expectation that agencies may adapt them to their specific needs, and the software and data specification have been created as open-source tools and written to allow for extensions. The early adopters will also serve as beta testers. This approach allows for additional software refinements to be made to the archive and information system, and/or for usability improvements to be built on top of the main software. For example, an agency may find that a graphical user interface would suit their needs better than editing CSV files in the Forecast Cards system, and it would be a logical extension to write such an interface that reads and writes Forecast Cards. Alternatively, it may be possible to streamline certain aspects of the data specification, or add to it, depending on the needs identified during the site visits. Task 4: Support and Community We expect implementation to be more successful with a plan for ongoing support, and propose to achieve this primarily through peer-to-peer support. Undoubtedly, users will encounter both issues and insights as they start to track and report forecast accuracy, and there is value in learning from each other as they do so. A natural first step in establishing such a community is through the early adopters, as discussed in Task 2. The site visits set up the initial process, but there must be some reason to continue the process after the site visit is done. Therefore, we propose that those early adopters be asked to participate in a peer exchange. This peer exchange would take place about a year after the site visits are complete, and provide the participants some opportunity to use the process and produce their first forecast accuracy summary report (Recommendation 3). The peer exchange would be modeled after the successful statewide modeling peer exchanges that have happened in recent years. Participants would present their initial results, and it would serve as a forum to share advice and work through any issues. Ensuring that this community support is sustainable relates in part to establishing an institutional home. Avenues are currently being explored to establish such an institutional home and take advantage of potential sources of pooled funding that could ease the burden on participating agencies. Participating agencies must individually implement recommendations 1 and 2; however, it may be more efficient for a single researcher to produce annual or biannual forecast accuracy update reports for several participating agencies. Such reports could use data pooled from across the participating agencies, and those agencies might “buy-in” to fund the joint analysis. An example involving a similar arrangement is the summary reports produced by Highway England’s Post-

III-E-6 Traffic Forecasting Accuracy Assessment Research Opening Project Evaluations (POPE). Such an effort could benefit from institutional support while allowing the participating agencies to avoid navigating the contracting requirements to obtain such support individually. A possible source of institutional support is the Zephyr Foundation, which was set up to facilitate shared resources and shared software within the travel forecasting community. We propose that the implementation team take the lead in identifying potential sources of support and present information about those sources to the participating agencies, along with any disclosures as might be needed to avoid potential concerns about conflicts of interest. Having reviewed the options, the participating agencies would make the decision about which path works best for them, and the implementation team would then facilitate making the arrangements with the chosen source.