Advanced Practices in Travel Forecasting (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

53 The lessons learned from the efforts to date were gleaned from the literature, documentation, and interviews with key model- ers and members of the topic panel. Each was asked, “If you had it to do over again, what would you do differently?,” a less confrontational manner than asking where they failed. Indeed, many had success stories to tell, lessons from which were as equally valuable as what not to do. Interestingly, most of these lessons relate primarily to institutional issues. It was believed that most of the key lessons from methodological and data standpoints were already adequately addressed in the litera- ture. What was lacking, most believed, was a digest of the practical issues associated with implementing such models in practice. This chapter attempts to fill that void. ASSESS THE CASE FOR ADVANCED MODELS The benefits of advanced modeling summarized in chapter three will be realized only if the questions asked by users of the model truly require them. Quite simply, if modelers are not being asked to model the complex issues of equity, pricing, sustainability, and other behavioral changes then most of the benefits of advanced models will not be apparent. In the past, travel demand models were typically used to estimate travel conditions under a variety of large-scale policy and invest- ment options, with levels of congestion and aggregate travel times as the principal outputs. Existing four-step models are arguably well-suited to producing credible and useful esti- mates of such travel. Although advanced models may improve the quality or accuracy of such forecasts in cases where travel behavior and choices are not expected to change significantly, it is difficult to quantify such improvements. Conversely, in cases where an advanced model is able to evaluate a policy that a traditional model cannot, the benefit is clear. For example, without a freight and commercial vehicle model, an agency could not effectively evaluate the potential benefit of truck-only express lanes serving a marine port. It quickly boils down to whether assumptions of relatively stable socioeconomic growth and transport policies and infra- structure are tenable to the agency. The agencies that have gotten the most out of advanced models are those for which such assumptions are not valid and those agencies where deci- sion makers are asking more expansive questions and demand- ing more comprehensive analyses. One interesting finding is that in addition to responding to the questions posed by decision makers, advanced mod- els can be used as a mechanism to prompt decision makers to ask more sophisticated questions and, if they see value in the information being provided, rely more heavily on the models themselves. In other words, the models in the plan- ning process can be used either proactively or reactively. The success of such endeavors depends on more than just the quality of the models themselves, but also on the role technical information plays in the planning process. Both SACOG and SFCTA, for example, have established cul- tures where planners and modelers are continually pushing each other to understand what insights can be gained from their suite of technical tools. VALUE OF A LONG-RANGE MODELING PLAN Most champions of advanced models began with a formal long-range plan for what they wanted to accomplish. This plan was used to educate decision makers and staff, define staffing and outsourcing requirements, and secure funding. It also set out the criteria for success, which in turn allowed for bench- marking of the various work efforts included within it. In some cases the plan was fairly simple, consisting of a few pages of broad overview and a roadmap, and in other cases it con- tained detailed work statements and resource scheduling for each step. The degree of detail appeared to be dictated by the requirements of the individual agency. The plan proved useful well beyond the initial justifica- tion and launch of the advanced modeling program. Many respondents used it repeatedly to remind agency executives where they were in the process and to demonstrate acceptable progress. Many believed that its value in retaining funding for advanced model development was more significant than for doing so initially, for it documented the dependencies of other programs on the current work. The plan was also used in the MPO certification process conducted by U.S.DOT, as well as for budgeting purposes within the agency and devel- opment of the Unified Planning Work Program. In some instances it was portrayed as a living document, with updates being made each year. It is notable that to date all of the successful models have been guided by such plans. Most of the champions that employed them were emphatic about their value, both for orga- nizing themselves and their staff and for selling the program. It was the most widely cited lesson learned, suggesting that its importance and value cannot be overstated. CHAPTER FIVE LESSONS LEARNED

IMPERATIVE OF THE CHAMPION Each advanced modeling success story examined was found to have one or more champions who spearheaded the adop- tion of the models within the agency. Moreover, that champion typically received strong support from one or more mentors or key executives within the agencies in order to succeed. A few had to build the relationship, but most benefited from a long-standing existing relationship. There was a surprising commonality among the champions: • The champions were motivated to expand their model- ing capabilities by their perceptions of changing analyt- ical requirements and the needs of decision makers. In most instances they reported that they believed that their existing models were not capable of answering the questions policymakers were likely to pose. • They shared the conviction that their modeling work would become less relevant to the agency and its sponsors if they could not be informed about current issues and proposals. • All started with an unwavering and clearly defined vision that required a year or longer to germinate before they could take the next steps. • Almost all of the champions had to find the money to fund the move toward advanced modeling. • They personally led the resulting work for several years after it was initiated, as the fledgling efforts often faced technical, funding, and institutional challenges that would have derailed them without the champion’s protection. In many instances the champion was not the technical leader of the advanced modeling effort, particularly in its ini- tial development. In most instances the champion hired con- sultants to handle many of those details. Rather, they focused on how the advanced models would be used in practice. Many helped policymakers decide “the right questions to ask,” and then translated those questions into model inputs and outputs. In some instances the champion was the consultant, which usually produced a less satisfactory outcome than when the champion was internal. In these cases the consultant persuaded the agency that the move to advanced models would better serve their interests than the status quo or incremental improve- ments. The consultant fulfilled all of the roles outlined earlier, except that they were the beneficiaries rather than providers of the funding. What was lacking in these instances was the close relationship with executives higher up in the client agency, such that their work was defined by contract rather than a closely shared long-term vision for the agency. In some instances an internal champion emerged to carry on the work of consultant-driven investments; however, in many cases the respondents lacked the same zeal and vision that characterized the agencies with internal champions. It is thought, but not yet apparent, that the need for the champion will diminish as advanced modeling practices 54 become established. The role of the champion would then blend with that of the leader of modeling at the agency. How- ever, the advanced modeling practices outlined in chapter two are all still evolving, which appears to reinforce the continued need for the champion for many years to come. Indeed, trans- forming the current generation of leaders in MPOs into such champions with the same vision, determination, and leadership skills is essential if these models are going to move into the mainstream of practice. ADVANCED MODELING REQUIRES MORE THAN MODELERS A great deal of emphasis was placed on finding, educating, equipping, and retaining the right people needed to succeed in advanced modeling. It was implied that there was a short- age of individuals entering the profession as well as of those already in it. However, almost all of the discussion focused on modelers without really defining who they are or how they are evolving. It is clear that modelers can be roughly divided into developers and users, with some working in both camps. However, in advanced modeling it is uncommon to find mod- elers with equal interest or aptitude for leading both activities. Neither the models reviewed nor interviews conducted brought fresh ideas to this important area. It was clear to most respondents that model development and software development were two quite different but highly complementary activities. Almost all agencies interviewed noted their unfulfilled need for software engineers with skills comparable to those of the model developers. The need for software specialists was apparent whether the model devel- opment was being done in-house or by consultants. How- ever, most agencies were unfamiliar with the software indus- try in general, including effective means for recruiting or career development needs and most productive practices for software engineering. Moreover, the need for software developers with skills relevant to high-performance scientific and engineering computing was viewed as being highly desir- able, but even harder to locate. It is conceivable that many advanced modeling efforts will be compromised by the lack of software implementations as sophisticated as the models themselves, making it essential that as much thought and effort goes into incorporating software engineering specialists as for the modelers themselves. The need for similar linkages with GIS specialists is readily apparent. However, most metropolitan planning agencies have already made substantial investments in GIS technology and staff. Excellent relationships between modelers and GIS spe- cialists were reported in those agencies interviewed. By con- trast, there were very few effective partnerships with infor- mation technology departments reported. In many instances information technology is viewed as an adversary rather than ally, a situation that can only hinder advanced modeling. There- fore, it is clear that establishing strong relationships that will lay the foundation for the use of appropriate hardware, operating

55 system, and programming practices needed for success in advanced modeling is essential. Without that foundation it is unlikely that advanced models can achieve their potential. CONTRACTING AND PROJECT MANAGEMENT FOR SUCCESS The approach used to contract and manage the implementa- tion of advanced models significantly affects its prospects for success, both technical and financial. It was apparent from this study that large, multi-year development projects are usually riskier than spending the same money on a series of smaller, incremental steps. Some of the more recent efforts have benefited from adopting the latter approach, which is becoming more widely used in other fields of engineering, and especially in software development. Agile development (AD) evolved in response to the frustration surrounding the high rate of failure of large software projects. The traditional approach—in both software engineering and advanced model development—had been multi-year mono- lithic development efforts where a scope of work drawn up at the outset became rigidly set in contract, and the remainder of the project revolved around implementing this “big design up front.” Not surprisingly, this approach did not work well when the domain knowledge was not mature enough and technological change was not slow enough to permit accurate estimation of the time and resources required for model development. Moreover, the definition of success is often fuzzy in research and development efforts such as this, where lessons learned during the research often shaped the work program (or at least could have). Advanced models surely embody these characteristics. AD seeks to overcome these barriers to successful delivery by emphasizing short, incremental development cycles. As with traditional approaches, sufficient time is devoted at the outset of the project to analyzing the problem and designing solutions. The best and most current ideas are still brought to bear on the problem. However, in a departure from most proj- ects, the user interface and elements of the user documentation are written up front, to focus the team on what is essential and usable from the client’s perspective. Understanding how the user expects to interact with the software or model becomes an important part of designing the solution. Equipped with such knowledge, the development team then proceeds to imple- ment the solution in short (typically one week to one month), incremental development cycles. “The simplest thing that can possibly work” (Occam’s Razor) becomes the first product, embodying little of the ultimately desired capabilities but having the advantage of placing interim software (and models in this case) in the users’ hands. Because software development is required to make advanced models operational (with the notable exception of the dynamic network models), many tenets of AD are directly applicable to these projects. The experience gained from the Oregon TLUMIP project illustrates the potential of AD. The overall process is shown in Figure 19. Listed across the top are the principal modeling capabilities originally specified for FIGURE 19 Example of the agile development process applied to advanced models (Source: Donnelly 2009).

the model. The first row shows the simplest implementation that would have worked, with progressively more sophisti- cated versions below it. Across the bottom are the ultimate desired capabilities. These are what would have been started from the outset using traditional approaches. Indeed, some of the components shown were attempted that way before shift- ing to an AD mindset. Adopting an AD approach allows each advance to be proven; if it does not lead to a significant increase in functionality it is not carried forward. Moreover, the devel- opment process can adapt as requirements change and experi- ence is gained with interim products. The successes using AD in Oregon, San Francisco, and for the ongoing statewide model development work in Maryland, as well as in other disciplines, suggest that it has great utility for advanced model development. Whether such an approach will prove necessary when more experience with such models is gained, or when established models are being transferred to new locations, remains unclear. It appears reasonable to assume that developers will have better success at estimating the cost, schedule, and potential pitfalls of mature products, making them better candidates for traditional contracting. However, for most development and implementation work without a clear track record of success it would appear that AD is a safer method of contracting. Cohn (2004) and Shore and Warden (2007) are recommended for readers wishing to acquire further insight. The principles of AD, known as the Agile Manifesto, can be found at http://www.agilemani festo.org. VALUE OF EDUCATION IS UNDERAPPRECIATED The need for education in advanced modeling was often mentioned during the interviews. The importance of contin- uously educating staff members, agency executives, policy- makers, and board members about modeling was emphasized several times. It was believed that if they understood the value and potential of modeling they would be more likely support it. Likewise, understanding what questions can be answered by a model would make them more likely to make informed use of it. Finally, it was believed that giving deci- sion makers an understanding of modeling would make them more receptive to requests for funding and support. Much of this education must be carried out at the local level, such that this job becomes the responsibility of the champion. Model- ing staff in agencies without a champion are at a disadvantage in this regard and therefore could take advantage of educa- tional materials prepared for this purpose by federal or TRB experts in the field. As already noted, the educational needs of modelers were brought up several times. These discussions ranged from need- ing to attract more graduate students into transportation and land use modeling, to improving the modeling skills of grad- uating planners and engineers, to the need for lifelong edu- cation for practicing modelers. These problems are chronic and long-standing, and have been cited many times in the 56 past to no effect (Weiner and Ducca 1996; Ben-Akiva and Bonsall 2004; Donnelly 2008). Absent concerted federal leadership there appear to be few prospects for innovation or advances in this area. DEBATE OVER OUTSOURCING VERSUS HOME GROWN REMAINS UNRESOLVED Most agencies believed it was helpful to outsource some development tasks to universities or consultants. Because model development happens only rarely in most cases it is not an efficient use of limited build staff capacity for build- ing advanced models, which requires skills in high demand. Moreover, even if staff with development skills are hired they typically lack the experience developers have gained implementing such models elsewhere. As one luminary in the field noted, “in learning the ropes they repeat most of the mistakes others have already learned from, only to finish with little opportunity to exploit that expensive education.” Without continued development work to keep them chal- lenged, these recipients of heavy investment often look for work elsewhere. The counter-argument is that outsourcing the develop- ment work leaves the MPO overly dependent on the devel- oper. In some cases this might be an acceptable outcome; however, in most instances is it clearly seen as detrimental. There is widespread consensus that all agencies should have staff with enough knowledge of the structure and internal workings of the models to permit their creative and compe- tent use and maintenance. Intimate knowledge of the model is necessary for users to adjust, debug, and extend the model, and to understand its key strengths and limitations. Such knowledge can rarely be imparted through a single or a few courses presented at the end of the model development effort. It cannot be assumed that several years’ worth of familiarity with the model can be distilled into a single week of training. Only a few cases were found where this tension had been adequately resolved: • The original activity-based model development work in Portland succeeded to the extent it did because of the close working relationship between the champion and the developers. The team met frequently, and the cham- pion made it a priority to be acquainted with the details of the on-going work. It was quickly acknowledged that this was possible only because the champion was shielded from day-to-day application issues by an equally quali- fied deputy. • The Oregon DOT dedicated one of its most experienced and capable modelers, at considerable cost in terms of operational capacity, to familiarize himself through testing and applying the model on an almost full-time basis over an extended period of time. However, it was only through this level of investment that the agency was able to usefully apply the models without consultant assistance.

57 In both instances the investment required from agency staff was considerable, in addition to the cost of outside developers. MPOs seeking to develop and deploy such models are well advised to include allowances for such investment in internal capabilities in addition to expenditures on outside help. It is argued that the adoption of AD practices will help overcome this tension. The rationale is that if outside but more experienced developers interact frequently with the ulti- mate users a more gradual and effective knowledge transfer will occur. Such a concept is facilitated by client participa- tion in reviews of the outcomes and the ability to use the interim working products from each cycle (from weeks to months each). This provides the opportunity for the ultimate users to engage in the process far earlier than traditional con- tracting allows. Although this has proven true in the software industry, its applicability to advanced modeling is only now becoming evident. MPOs face some obstacles that other AD adherents do not, in particular that developers are often located in distant cities, making their availability for frequent meetings questionable. One obvious solution is to use consultants or academics in a purely advisory and quality oversight role. In such an arrangement they would provide formal training and coach- ing to agency staff responsible for doing the actual work involved. This would be practical only when the academic or consultant is located nearby, as a high amount of interaction would be required at the outset. No example of this contract- ing model was found, although the city of Calgary and the University of Calgary have used such arrangements effec- tively for smaller projects.