Return on Investment in Transportation Asset Management Systems and Practices (2018)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Drawing from current literature and the case studies conducted for the project, the research team developed a framework for conducting an ROI analysis on a new or upgraded TAM system or process. The most information is available—and systems most often are adopted—for pave- ments and bridges, so the framework focuses on improvements to TAM systems for these assets. Many of the framework’s methodological elements can be generalized to other classes of assets. The framework’s theoretical underpinnings and approach underlie the guidance presented in Chapter 4, which offers more detailed guidance for practitioners, complete with a step-by- step approach to conducting an ROI analysis on a recent or proposed TAM investment or improvement. Both this framework and the more detailed guidance are designed to guide ROI estimations at a reasonable level of effort, without requiring significant new data collection and research. The benefits side of the analysis focuses on those benefits supported by evidence in the lit- erature. Agencies also can use the framework to measure the actual returns realized from past investments (relative to returns that could have been realized under ideal circumstances) or to estimate the potential returns from future, prospective investments. Several basic principles guided the development of the ROI estimation framework, as follows: • The benefits and costs should be expressed in dollar terms whenever possible, so that the com- parative value of costs and benefits can be better understood. Benefits and costs that cannot be quantified (intangibles) are assessed qualitatively. • All benefits and costs are to be measured relative to a counterfactual base case of no invest- ment. The exact parameters of the base case can vary by agency, but once defined, the base case must remain consistent throughout each analysis. • Future benefits and costs are to be discounted to their present values to allow comparisons across time. • Benefits and costs should be considered from a broad societal perspective; however, benefits are to be estimated separately for agencies, asset users, and the general public. • The value of investments will be assessed through comparisons of total benefits to total costs, and through the estimation of net present value (NPV), benefit-cost (B/C) ratio, or the internal rate of return (IRR). The ROI estimation framework primarily uses language and themes associated with the establishment or improvement of the most commonly found TAM systems, such as: • Pavement management systems (PMS), • Bridge management systems (BMS), and • Maintenance management systems (MMS). C H A P T E R 2 Framework for Estimating ROI

4 Return on Investment in Transportation Asset Management Systems and Practices The framework also incorporates the most common types of TAM system investments, such as: • Purchases of an initial asset management system, • Improvements to an existing asset management system, or • Enhancement of an agency’s data collection and reporting methods. This framework presented in this chapter is broad enough to engage potential users at a variety of stages in adopting or adjusting TAM practices and systems, and with different levels of com- plexity. It lays out a guidance approach that can be used to estimate the returns from upgrades to existing systems, initial adoptions of systems, or both, and by agencies with state-of-the-art systems and significant asset management experience or by agencies with limited systems and experience. Using a Benefit-Cost Analysis Approach ROI measures the amount of financial return on an investment relative to the investment’s cost. The returns may be a single payment or a stream of payments. ROI often is used in capital markets to compare the returns from multiple investments. In the transportation field, the returns on any investment are likely to be interpreted more broadly. The Contractor’s Final Report to NCHRP Project 8-36, Task 62, “Best Practice Methodology for Calculating Return on Investment for Transportation Programs and Projects,” (2008), notes that ROI analysis for publicly funded projects is best understood as benefit-cost analysis (BCA). Concurring with that assessment, the project team built the ROI estimation framework using a BCA approach. BCA is a comparative method in which all the costs and benefits to society (which includes agencies, drivers, residents, etc.), are estimated and then assigned dollar values, so that total costs can be compared to total benefits over a period of time. Figure 2-1 graphically presents a key aspect of BCA, that the costs of an investment are “weighed” against its benefits. In recent years, increasing numbers of programs compare or prioritize multiple potential transportation investments using a BCA approach, including several federal discretionary funding programs such as Transportation Investment Generating Economic Recovery (TIGER). BCA can be used to measure actual returns realized from past (retrospective) investments against their costs or to estimate potential returns from future (prospective) investments against Figure 2-1. BCA of a TAM investment: Weighing costs and benefits.

Framework for Estimating ROI 5 their likely costs. The underlying approach to estimating the ROI is the same for both retrospective and prospective evaluations; however, some differences exist: • In prospective evaluations using traditional BCA, only current conditions are observed. Pro- jections (e.g., future agency costs, pavement conditions, or vehicle operating costs) are needed for both the base case and the investment case. • In retrospective evaluations, both the current (even starting) conditions and the investment case are observable. Estimates are still needed for the base case. For these evaluations, the analyst must predict what would have happened to agency costs, pavement conditions, and vehicle operating costs had the project not been implemented. Although retrospective evaluations are traditionally considered more accurate and reliable than prospective evaluations, both types of evaluation rely on some form of forecasting. The use of forecasts introduces a degree of uncertainty in the estimation (which is addressed later in this chapter, as well as in the guidance). Definition of Base Case and Investment Case ROI or BCA estimate the benefits and costs of an investment by comparing two states of the world: a world with the investment, and a world without the investment. The state of the world without the investment often is called the base case or the no build, whereas the version of the world with the investment is called the investment case, the project case, or for construction proj- ects, the build case or build scenario. The ROI estimation framework developed in this report uses the terms base case and investment case. The base case against which a project is assessed must be realistic. This means the base case normally does not reflect doing absolutely nothing, but rather reflects what would occur “but for” the investment being analyzed. For example, in the BCA guidelines for TIGER discretionary grant applications, the U.S.DOT (2015) notes that the baseline should assume a continuation of reasonable and sound management practices1 and that “a baseline scenario in which the owner of the facility does no maintenance on the facility and ignores traffic problems and maintenance is not realistic. . . . ”2 The base case is important on both the costs side and the benefits side of the equation. Ongo- ing costs that would be incurred with or without the investment must be incorporated into the analysis, as must any associated benefits. For instance, a base case with continuation of an existing (pre-TAM) approach to maintenance must include the ongoing costs of the existing approach (e.g., regular maintenance costs and any improvements to assets from that maintenance). If a TAM system leads to more efficient use of agency resources (e.g., staff, expenditures for repair/ rehabilitation), then the agency may be able to address needs associated with additional assets, leading in turn to an overall improvement in asset condition and an improvement in user travel. When very large or complex capital investments are evaluated through BCA, the investment case may be compared to a base case that includes both low-cost capital solutions (e.g., improved signage or ramp metering) and productive non-capital solutions (e.g., demand management strategies).3 The difference between the base case and investment case in such situations may be considered to be a range of values or multiple possible results. An additional complication for the evaluation of TAM investments is that, ideally, existing and future conditions should be determined for both (1) TAM systems and agency operations, and (2) the transportation assets (e.g., pavement, bridges, and maintenance items) potentially affected by these systems and operations. For example, current transportation asset conditions will affect the expected level of benefits that accrue to the general public.

6 Return on Investment in Transportation Asset Management Systems and Practices Another complicating factor is that the scale of benefits from similar TAM investments in differ- ent situations will vary depending on the context and uses of each system (e.g., the system’s com- plexity and coverage, the general level of TAM systems and practices already in place, and even the existing asset condition at each location). A new or upgraded TAM system has greater potential to generate a high return if it is applied to a variety of asset conditions. The existing overall asset condi- tion matters because few improvements are available for assets that are generally in a good state of repair, whereas nearly any maintenance decision will result in an improvement if assets are primar- ily in a state of poor repair. Opportunities for optimization vary with the overall state of the assets. The review of existing literature on TAM ROI found that the assessments generally used a com- parison of “business as usual” (existing or traditional methods and processes) to a state in which the TAM solution is deployed and used, reinforcing the appropriateness of this approach for a range of TAM systems and approaches. Several important example scenarios are summarized in Table 2-1. The investment case includes the TAM changes being made, such as a new or upgraded system or a change in practices, as well as its costs (initial and ongoing) and its impacts on the agency, the assets, and, potentially, the asset users. The framework focuses on the implementation of TAM Study Investment Case Base Case Babinski et al. (2012) With GIS (output level and time spent producing output using GIS) Without GIS (output level and time spent producing output before implementation) Cambridge Systematics & Meyer (2007) After implementation of system preservation strategy Before implementation of system preservation strategy De la Garza et al. (2011) With IP-S2 mobile mapping technology Traditional manual inspection & data collection Dye Management Group (2014) With remote data collection technologies (mobile imaging, mobile imaging with LiDAR, aerial imaging with LiDAR) Manual data collection techniques Hoekstra & Breyer (2011) With Statewide Multi Level Linear Referencing System (MLLRS), baseline and two optional functional elements Without MLLRS, but with LRS Kin et al. (2011) With mobile LiDAR technology (7 deployment options) Existing data collection methods & processes (e.g., personnel deployed in field) McNeil & Mizusawa (2008) With PMS (simulation of PMS optimization strategy) Without PMS (simulation of “worst first” strategy) McNeil et al. (2011) With HERS-ST (simulation with HERS- ST prediction and BCA procedures) Without HERS-ST (simulation with treatments selected based on PSR by highway class) Nokes et al. (2011) With Heavy Vehicle Simulator (HVS) testing of five pavement rehabilitation alternatives Without HVS testing Schiffer (2006) With automated asset inventory system (assessed through pilot study) Existing, manual inventory methods Smadi (2004) With PMS (simulation of PMS optimization strategy) Without PMS (actual expenses before implementation) Vasquez et al. (2010) Partial pavement management (preventative treatment 8 years after newly constructed); Full pavement management (partial plus rehabilitation treatment 4 years later) No pavement management (complete base and pavement replacement after pavement failure) Ye et al. (2009) With Winter Maintenance Decision- Support System, MDSS (simulation of recommended practices under two scenarios) Without Winter MDSS (simulation of traditional practices) Table 2-1. Example scenarios evaluated in the literature.

Framework for Estimating ROI 7 systems rather than on TAM practices that would be adopted independently of new or improved hardware or software solutions, because it is usually much harder to measure the impacts of changes in practices than the introduction of a new or upgraded data system. Nevertheless, the framework can support analysis of the benefits and costs associated with collecting additional asset inventory and condition data, or changing the methods for collecting or processing these data (e.g., through the use of automated data collection techniques, or the integration of GIS capabilities), provided the data can be gathered on any incremental changes that arise from those activities along with their associated costs. Change in Agency Focus due to TAM Practices In most situations, the base case and investment case for TAM system investments will likely fall somewhere between these extremes: • Base case: Use of emergency, reactive decision-support systems, leading to large and costly maintenance projects and “big ticket items”; and • Investment case: Focus on asset preservation, with smaller projects identified through the use of TAM systems and processes. In other words, the initial deployment of TAM systems and improved processes can help agencies switch from a reactive, failure-avoidance focus to a preservation focus. Improving TAM systems and processes will involve less dramatic differences between the base case and the invest- ment case, but the general concept remains the same. Improvements in data quality can similarly show differences in the outcomes of asset management decisions by simulating optimal and less-than-ideal conditions. When the TAM Investment Affects Agency Decision Making Some of the benefits from a new or improved TAM system depend on the extent to which agency decisions are based on the output of the system. In particular, benefits to asset users (e.g., vehicle operating cost savings from smoother pavements, or reduced travel times due to avoided bridge closures and detours) will depend on whether the TAM system output has been factored into agency decisions. Ultimately, the returns from the potential scenarios that can be evaluated reflect both the level of sophistication in the TAM system being deployed and the degree to which the agency adheres to the recommendations offered by the TAM system. At the extremes, these scenarios range from the complete absence of TAM systems and methods (worst scenario yielding the worst ROI), to the deployment of a state-of-the-art TAM system combined with optimal utilization (best scenario yielding the best ROI). Other combinations are also possible (see Figure 2-2). It bears repeating that the variety of transportation asset conditions also affects the level of benefits to be expected from TAM. The x axis shown in Figure 2-2 could be relabeled “Asset Conditions” and the y axis “Diversity of Asset Conditions” with similar results. When consid- ered together with system capacity and use of system outputs, the condition and diversity of assets adds another dimension that underscores the complexity of the problem at hand. The level of sophistication of a TAM system can be expected to have a significant effect on the magnitude of potential benefits and costs. This level of sophistication can be defined in terms of: • Percentage of assets covered, • Percentage of agency needs addressed, • Number of potential treatments, or • Other agency-specific measures.

8 Return on Investment in Transportation Asset Management Systems and Practices If the analyst believes the impact of the analysis will be affected by the scope or scale of the TAM system, by changes in the TAM system or practices, or by the degree to which the agency already uses TAM systems and recommendations, the analyst can scale the projections up or down during the analysis. Broader scope and/or complexity will likely be associated with greater impacts, whereas greater existing use of TAM systems or practices will likely reduce the impact of additional TAM investments. Benefit and Cost Categories TAM investments have the potential to produce direct effects on agencies and asset users as well as indirect effects on the economy. Direct effects typically involve changes or impacts experienced by agencies, asset users, or the communities most directly affected by the use of the transportation asset(s) examined. These effects can include reductions in staff time and maintenance expenditures, avoidance of increases in vehicle operating costs and travel time, and reductions in emissions and noise. The indirect economic effects from TAM investments can be described as those changes in eco- nomic activities that arise from the direct effects. As the U.S. Government Accountability Office (GAO) stated in a 2005 report on highway and transit investments, “lowering transportation costs for users and improving access to goods and services enables new and increased economic System U se o f S ys te m O ut pu t Figure 2-2. Potential scenarios for evaluation and expected returns.

Framework for Estimating ROI 9 and social activity.”4 Indirect economic effects can be measured in terms of changes in jobs, tax revenue, wages, or work output (productivity). It is important to remember that indirect economic effects are a function of the direct effects. They are, in a sense, just a restatement of the impacts already measured in direct effects. Several texts, including the FHWA’s Economic Analysis Primer (2003), discuss the differences between BCA and economic impact analysis (EIA) and how inclusion of economic impacts in BCA could result in counting the same effects twice.5 The ROI estimation framework developed in this study therefore includes only direct effects; indirect and local economic effects are not included. Multiple Recipients of Benefits and Costs Costs fall into one of two categories: the agency’s investment costs (capital and recurring costs) and any potential user costs which are sometimes termed disbenefits. Disbenefits can include tem- porary increases in travel time (such as those that occur during construction activity) and other inconveniences that arise from the implementation of agency decisions based on TAM systems or improved data collection. The majority of the existing research on TAM effects discusses agency impacts in terms of changes in operations and maintenance cost or changes in data collection, processing cost, and analysis cost. Some studies also acknowledge the existence of many potential direct effects to the asset users, and the relation of other effects to the community as a whole (e.g., changes to emissions). According to the literature review, the benefits of TAM investments accrue across three stakeholder groups: • The transportation agency, • Asset users, and • The general public. In the ROI estimation framework, benefits and costs are measured from a broad societal perspec- tive, but separate estimates can be made to examine benefits and costs in relation to each of the three groups. In particular, estimates of benefits and costs to the implementing transportation agency are beneficial because they allow estimation of returns to the agency itself. Benefits to asset users (e.g., travel time savings) and to the general public (e.g., health benefits of reduced vehicular emissions) also are valuable to measure, but they require additional steps to estimate. Figure 2-3 provides an overview of the benefit and cost categories to be considered in the ROI estimation framework. Agency Benefits Asset-User Benefits Benefits to General Public Capital Costs ($) Recurring Costs ($) Total Benefits ($) DiscountRate (%) Total Costs ($) ROI Estimates ($) Figure 2-3. Overview of benefit and cost categories.

10 Return on Investment in Transportation Asset Management Systems and Practices With a PMS, the type and magnitude of user benefits will depend in part on the impact of TAM on the likelihood of structural versus functional failure. A structural failure that renders a pavement section or bridge incapable of sustaining any traffic load imposes a higher impact on users than does a functional pavement failure that causes a degree of surface roughness and consequent discomfort to the road’s users. Taxonomy of Benefits As a first step toward building an ROI estimation framework for assessing the benefits of TAM investments, the research team constructed a taxonomy of potential benefits based on the find- ings from the literature review. Table 2-2 shows benefits organized by the three principal stakeholder groups to whom the benefits accrue. Some of the benefits identified, such as vehicle operating cost savings or travel time savings, can be expressed in dollar terms. Benefits that cannot be quantified (intangibles) must be assessed qualitatively. For example, it may be almost impossible to place a dollar value on the enhanced reputation an agency experiences through improved information sharing, but this benefit could be extremely valuable to the agency. Taxonomy of Implementation Costs Investment costs are required inputs to any ROI assessment and should include costs over the entire analysis period. Table 2-3 lists costs that may occur over the life cycle of TAM invest- ments. These costs are broken into two primary categories: (1) non-recurring costs, which may be initial investments or renewal investments, and (2) recurring costs that are part of operating and maintaining the TAM investment (rather than the assets being managed). Direct and Indirect Agency Cost Savings Staff time savings from improved data collection and accessibility Cost savings from the optimization of investment strategies Lower costs from reductions in failure risks for critical assets (e.g., bridges) Avoided outlays for legacy systems, including hardware maintenance and software updates Enhanced reputation and level of public trust gained through information sharing Delayed capital expenditures due to increased asset life (residual value of assets) Reduced worker safety costs (due to bundling of projects) Residual value (remaining asset value at end of analysis period) User Cost Savings Vehicle operating cost savings (e.g., reduced wear and tear, reduced fuel consumption) from smoother pavements or more direct routing (e.g., with enhanced bridge availability through improved maintenance and a reduction in postings) Travel time savings (e.g., reduced work zone delays, reduced time spent on detours) Savings from accelerated improvements to transportation system maintenance, rehabilitation, or capacity that reflect timely agency decisions regarding asset management Safety benefits (e.g., briefer exposure to work zones, alternate/unfamiliar routes, temporary traffic pattern changes; overall improved safety of transportation infrastructure being used) Benefits to the General Public (Social Benefits) Reduced emissions (e.g., from smoother pavements or more direct routing) Reduced noise generation Table 2-2. Potential benefits of TAM investments by stakeholder group.

Framework for Estimating ROI 11 In establishing the analysis period, it is important to keep in mind the distinction between the useful life of assets, such as pavement or bridges, compared to the useful life of TAM investments. Pavement improvements may last 10 to 30 years, depending on the treatment, and the life cycle of a bridge may be 75 years. In contrast, the useful life of a TAM system could be much shorter—perhaps 5 to 10 years. This is a short time period over which to see meaningful changes in asset condition and the benefits that result from improved conditions. The ROI analysis of TAM systems and practices should consider a long period (e.g., 10 to 20 years), which means that the life cycle costs need to take into account renewal costs, such as software upgrades and new acquisition costs at appropriate intervals. The residual value at the end of the analysis period can be calculated to account for benefits extending beyond the analysis period. Methods for Estimation of Benefits The exact methods and assumptions used in the estimation of benefits will depend on the characteristics of the TAM systems being evaluated, the context in which they are deployed, and data availability. Structure and logic (S&L) diagrams can illustrate how a benefit metric is calculated. An S&L diagram graphically represents an equation; each box stands for a variable or parameter (e.g., input, model coefficient, intermediate output, and final output), and links between boxes stand for operations (e.g., add, multiply or divide). Figure 2-4 shows a generic S&L diagram. Methods for Estimation of Agency Benefits The methods used to estimate agency benefits apply to both retrospective and prospective evaluations. The statistical techniques used in the measurement of performance and the quan- tification of input values will be described later in this chapter. Agency cost savings—arguably the most common manifestation of agency benefits—are esti- mated as the difference between agency costs in the base case and agency costs in the investment case. If new activities or responsibilities are created in the investment case, they are treated as disbenefits or incremental costs. These disbenefits need to be balanced with the value of any benefits associated with the new activities. Non-Recurring Costs Hardware and software acquisition Installation Training Decommissioning Shift in investments (e.g., delay in some investments to perform additional preservation or other work) Recurring Costs Maintenance and repair Operating expenses Software maintenance costs Software updates Data collection and data analysis cost Table 2-3. Life cycle costs of TAM investments.

12 Return on Investment in Transportation Asset Management Systems and Practices If the investment leads to an increase in productivity (more units produced per unit of time, or lower agency cost per unit) and an increase in total output such that total agency expenses remain more or less the same, both impacts will be highlighted and estimated separately. Dif- ferences in productivity are estimated as the difference in costs between the base case and the investment case while holding agency output constant at the base-case level. The value of any added output is quantified separately. A related consideration is the potential labor cost savings brought about by automating TAM processes and new TAM systems. Whether or not these lead to actual cost savings for the agency (i.e., reductions in expenditures), the full value of the labor productivity gains should be counted as a benefit in the ROI assessment. The rationale for this is that labor resources (inside or outside the agency) are being freed by the investment to accomplish other tasks of equal or higher value. Methods for Estimation of User Benefits Estimating user benefits requires intermediate output generated by the TAM system (e.g., asset condition and asset usage data) and the use of algorithms built into the asset management Figure 2-4. S&L diagram concept.

Framework for Estimating ROI 13 systems, software, or spreadsheet. Four broad categories of input values are needed to character- ize the base case and investment case: 1. Measures of asset condition (e.g., pavement smoothness) for the entire system or by facility; 2. Traffic volumes affected by the ROI investment; 3. Measures of transportation system performance, such as travel times, average vehicle speed, or level of service; and 4. Other characteristics of the transportation system that might affect generalized travel costs or travel demand, such as the length of a detour in the case of a bridge failure. Traffic volume data may come from the TAM system or from an outside source (e.g., a travel demand model). If possible, the data should be distributed by trip purpose (e.g., personal or busi- ness), time of day (e.g., peak and off peak), and/or vehicle type (e.g., cars and trucks). The geo- graphic area and assets included need to be defined precisely. Measures of transportation system performance and other transportation system characteristics also may come from the TAM system, but it also may be necessary for the analyst to consult one or more outside models or sources. Traffic volume data may come from the TAM system or from an outside source (e.g., a travel demand model). If possible, the data should be distributed by trip purpose (e.g., personal or busi- ness), time of day (e.g., peak and off peak), and/or vehicle type (e.g., cars and trucks). The geo- graphic area and assets included need to be defined precisely. Measures of transportation system performance and other transportation system characteristics also may come from the TAM system, but it also may be necessary for the analyst to consult one or more outside models or sources. Figure 2-5 shows an S&L diagram for the estimation of vehicle operating cost savings resulting from smoother pavements. User benefits also can be estimated using the ROI Tool discussed in Chapter 5 of this report. Measurement of Performance and Quantification of Input Values Several methods can be used to quantify input values for use in the estimation of TAM ben- efits and the assessment of ROI. The descriptions of these methods presented in this chapter are relatively brief, and are based in large part on the findings of the literature review. Cost and benefit metrics typically have two components: (1) a quantity component, expressed in physical units (e.g., annual staff hours spent processing pavement condition data, hours of congestion, gallons of fuel consumed); and (2) a value component, expressed in dollars (or dollar-equivalent) per unit of measurement (e.g., agency expenses per staff hour, value of time per hour, average fuel cost per gallon). Overview of Key Methods The literature review identified several methods that can be used to model and predict the values of various performance indicators, depending on the quality and quantity of the infor- mation at hand. Changes in the values of performance indicators between the base case and the investment case are important inputs in the estimation of TAM benefits and costs. Figure 2-6 summarizes the methodology used by the research team for the measurement of impacts needed to estimate benefits for the BCA. In order to go from the “Inputs” stage to the “Outcomes” stage, one needs analytical methods to quantify the impacts of TAM in terms of asset conditions, data collection costs, and asset maintenance and rehabilitation costs. These impacts can be translated into agency or user benefits, depending on the nature of the item

14 Return on Investment in Transportation Asset Management Systems and Practices Figure 2-5. Sample S&L diagram for vehicle operating costs. being quantified. Care must be taken in deciding which method to use for a particular agency, as these methods may not all accommodate limitations associated with data scarcity, level of adherence to TAM policies, or external factors which can alter the effects of improved asset management. Based on the literature review, at least four types of methods are available to quantify the effects of TAM systems and processes: • Simulations, • Controlled field experiments, • Time series analyses, and • Breakeven analyses. Simulations Simulation methods enable researchers to model the “what-ifs” of a system or process. They can be used to estimate the impacts of a TAM system by simulating the asset conditions and agency costs under the base case and the investment case scenarios, and they can make more

Framework for Estimating ROI 15 refined projections than many other methods. In most situations, the TAM system itself offers the means to simulate outcomes under varying asset conditions and treatment assumptions. For example, “With TAM Investment” (investment case) and “Without TAM Investment” (base case) can be simulated so that both produce similar system-wide measures of asset conditions, in which case the relative merit of the two scenarios can be assessed in terms of the agency costs required. Conversely, the scenarios can be given the same budgets, in which case their relative merit can be assessed in terms of system condition and user costs. For a retrospective analysis, another method would be to use actual historical data for the investment case and use the TAM system to simulate the base case. This approach adds another element to the ROI analysis: the error rate of the TAM system in predicting actual conditions. The error rate can be tested in the case studies by making the comparison using historical data with the TAM system compared to using the TAM system only. Table 2-1 included three studies that evaluated TAM systems using simulation techniques.6 These research examples demonstrate the versatility of simulation techniques when assessing baseline asset conditions to one or more alternative scenarios. A criticism of simulation models is that the same decision criteria used by the TAM sys- tems to select asset treatments are used to assess the performance of those treatments or tools relative to current practices. The concern is that, even if the TAM system’s prediction models and prioritization procedures are significantly flawed, the tools will score better than current practice because the flawed elements are used to keep score. This concern may not be a major issue, however, provided the prediction models have been independently verified. Figure 2-6. Inputs, outputs, and outcomes for the measurement of ROI.

16 Return on Investment in Transportation Asset Management Systems and Practices Controlled Field Experiments The literature review included several studies in which controlled field experiments were per- formed to quantify the benefits of TAM actions, although not all the results of these experiments may be applicable as direct inputs into an ROI evaluation. More often than not, data collected from controlled field experiments serve as input when developing an asset management system.7 Typically, controlled field experiments focus on a specific component of a TAM system (e.g., data collection for a certain type of asset or for barcode inventory systems). A transportation agency may be able to use controlled field experiments (1) when the agency follows TAM system recom- mendations for a portion of its system and uses business-as-usual practices for the remainder, or (2) when the agency’s neighboring jurisdictions follow different asset management practices and data from the two methods can be compared. After controlling for external factors over a suffi- cient period of time, differences in performance indicators can be attributed to the TAM system. Time Series Analysis Of all the quantitative methods available to track changes in asset conditions or agency expenses, time series modeling is one of the easiest to understand. This method is ideal if an agency wants to document the benefits associated with ROI actions that have already been implemented for many years (e.g., 10 years or more). Changes in performance can be displayed with time series charts, and trend analysis or hypothesis testing can be used to assess the impacts of TAM implementation on a given performance measure. No such analyses were found in the literature review, however, perhaps due to a lack of examples in which sufficiently extensive data was available to support time series analysis. Breakeven Analysis Breakeven analysis is useful when most of the benefits of a TAM action cannot easily be quan- tified. In breakeven analysis, the benefits that would be required to cover the costs of a TAM action are calculated and related to the size of the system that the action would affect. The project team reviewed two reports that demonstrated the use of breakeven analysis (see Appendix A for more detail).8 Although breakeven analysis may provide the justification needed to validate a policy or investment decision when more exact quantification methods are not possible, it is not a performance measurement method. Application of Methods to Prospective and Retrospective Evaluations The research team recognizes that each of the above methods has its own advantages and disadvantages with respect to producing quantifiable inputs for ROI evaluation. When data, scope, or budget limitations prevent the application of these methods, techniques such as benefit transfer or expert assessment can be used. In addition, although simulation, benefit transfer, or expert assessment may be used in both prospective and retrospective evaluations, time series analysis and controlled field experiments are applicable only in retrospective evaluations. Table 2-4 summarizes the advantages and disadvantages of the methods described. ROI Assessment and Reporting This section describes standard ROI concepts, such as the distribution of benefits and costs over time, the use of a discount rate, and the reporting of ROI results using standard measures. A more detailed treatment of these issues can be found in standard texts on BCA or engineering economic analysis, such as those by Mishan and Quah (2007)9 and de Neufville (1990)10.

Framework for Estimating ROI 17 Distribution of Benefits and Costs over Time When conducting a BCA, it is important to consider the distribution of benefits and costs over time. Investments, such as new or improved TAM systems or revised TAM practices, incur upfront capital costs and ongoing operating costs associated with licensing, software updates, training, and data collection. Renewal costs (e.g., purchases of new computer hardware or other system parts) also may be associated with refreshing TAM systems as they reach the end of their product life cycles. On the other hand, the benefits of TAM systems and practices start to accrue only after the upfront deployment costs have been paid. This means the stream of benefits and costs are not evenly distributed over time. Ramp-up of Benefits Immediately following the initial TAM investment, benefits may be lower than optimal and lower than what might be expected given the need for equipment rollout, staff training, and other factors. It may take a while for an agency to trust a TAM system and begin using the sys- tem to make decisions. In addition, a backlog in already-committed projects may account for programmed maintenance and rehabilitation funding. As a result, the estimation of benefits needs to account for any potential ramp-up of benefits or investment productivity following the Technique Advantages Disadvantages Application Simulation Can control circumstances Multiple choices of performance measures Available at most agencies Flexibility in design Error in TAM tools is unknown Unclear if total benefits estimates are over or under true value Because of flexibility, simulation techniques can be used for either prospective or retrospective evaluations Controlled Field Experiments Use of scientific method to validate transportation improvements or data collection processes—objective results Time and resources needed to implement experiment Difficulties associated with identifying control group Retrospectively (if experiment was set up at the time of the investment); or prospectively, through pilot program Time Series Controls for multiple factors Relatively easy to implement if data are available Requires techniques to control for changes to operations and policy over time at a given agency Yet to be implemented in domain In retrospective evaluations only Breakeven Analysis Can be used in situations where direct quantification of effects is not possible Helps assess where investment is likely to be beneficial No performance measurement per se Cannot be used as basis for estimating returns In retrospective or prospective evaluations Table 2-4. Summary of quantitative methods.

18 Return on Investment in Transportation Asset Management Systems and Practices initial deployment period. Some agencies may find it useful to report average annual benefits in “steady state” after the initial ramp-up of benefits occurs. Other Changes in Benefits over Time ROI analysis also needs to account for the fact that benefits will change over the life cycle of the investment. A common approach in BCA for transportation projects is to estimate benefits for a base year (when the initial investment is made) and for a forecast year (at the end of the project life cycle). Benefits for intervening years are then interpolated between these two years. Another approach is to estimate future benefits from the initial year (or an average year) using a growth factor. These approaches can seriously underestimate or overestimate benefits because the ben- efits tend to be non-linear functions of other factors, such as vehicle-miles traveled (VMT) and vehicle speeds. It is more accurate to interpolate the input variables and estimate the benefits for each year. Unfortunately, none of these approaches works well for estimating the ROI of TAM invest- ments. One benefit of TAM investments is savings in agency costs for the maintenance and rehabilitation of assets over time. The maintenance and rehabilitation costs are a function of the inventory and starting conditions of the assets at the time of the TAM investment. As shown in Figure 2-7, an agency that has many assets simultaneously nearing the end of their asset life has a very different cost profile from an agency that has an even distribution of rehabilitation needs over time. This difference in profile affects the baseline conditions and the range of treatments that can be recommended for asset management. All else being equal, a TAM system is most likely to be beneficial to an agency with a range of rehabilitation needs somewhere in between the two extremes illustrated in Figure 2-7. ROI analysis for TAM investments also needs to take into account the deterioration of assets over time, which will affect the timing and magnitude of treatments suggested by TAM systems. Factors (e.g., weather and usage) that influence the pace of deterioration are incorporated into TAM systems. The benefits cannot be estimated simply for a few years and then expanded (through interpolation or extrapolation) to cover additional years in the analysis period, and this complexity explains why simulation analysis using TAM systems is compelling. Discounting Discounting plays an important role in ROI analysis to convert the uneven distribution of benefits and costs over time into values that are comparable. De Neufville (1990) describes the need for discounting as a dollar today being worth more than a dollar in the future because it can Figure 2-7. Agencies with evenly distributed and unevenly distributed rehabilitation needs.

Framework for Estimating ROI 19 be used productively in the intervening time. A more intuitive way to think about discounting is that people place a greater value on having money (or benefits) now than in the future. Given the choice to have an ice cream cone today or wait until next week, most children will choose to have the ice cream today. Discounting accounts for the lowering of the value of “next week’s” ice cream cone. In this way, discounting puts future values in present terms. It is helpful to distinguish between inflation and discounting, and also between the nominal discount rate and the real discount rate: • Inflation accounts for how the value of money changes over time; • Discounting accounts for people’s preferences to consume today rather than in the future; • A nominal discount rate takes into account both inflation and people’s preference to consume today; and • A real discount rate discounts future benefits even after inflation is taken into account (i.e., after all benefits are expressed in current year dollars). A real discount rate tends to be lower than a nominal discount rate. Several sources provide guidance on setting a discount rate for BCA or ROI evaluation. The U.S. Office of Management and Budget (OMB) recommends evaluating benefits using two dis- count rates depending on the type of investment being analyzed: 7% for analysis of new invest- ments and regulations; and the Treasury’s borrowing rate for analyses of cost-effectiveness, lease-purchase, internal government investment, and asset sales analyses.11 The OMB’s latest guidance estimates the real 20-year borrowing rate to be 1.2%, reflecting the historically low discount rate environment at present. A discount rate of 4% was used for the case studies and pilot conducted for this project. The justification for this selection was as follows: • Investments in asset management systems and processes are more akin to cost-effectiveness or internal investments than to investments in new infrastructure. Thus, based strictly on the OMB guidance, these investments would be analyzed using the Treasury’s borrowing rate of 1.2%. • In performing the case studies and pilot, the research team sought to avoid overstating the potential benefits of investing in an asset management system or process improvement. This caution argued for using a discount rate greater than 1.2%, given that the current borrowing rate (during this research) was at a low point. • Historically, real discount rates have been 3% to 5%. Because the case studies look backward to historic investments, a rate in this range would be more representative of the decision- making environment at the time the case study investments were made, so the researchers elected to use a 4% discount rate. • Although the pilot is forward looking, it also uses a 4% discount rate to avoid overstating potential benefits and to be consistent with the case studies. Some states have their own recommended discount rates for analyses; other states follow the OMB guidelines. Regardless of the discount rate that is used, it is often useful to test alternative values for the discount rate by performing a sensitivity analysis. ROI Measures Once the ROI analysis has been conducted, several measures are available for summarizing the results. All of the measures compare the benefits and costs estimated over the TAM invest- ment life cycle. The interpretation and use of the measures will vary, but all provide a summary of whether or not the investment is worthwhile.

20 Return on Investment in Transportation Asset Management Systems and Practices Net present value (NPV): Perhaps the easiest measure to understand, NPV shows what the TAM investment is currently “worth.” It is calculated as the present value of the benefits (i.e., all benefits discounted to the present) minus the present value of the costs (including capital, main- tenance, and operating costs discounted to the present). NPV is not scaled to the size of the invest- ment. A $5 million NPV is better than a $2 million NPV, even if the former required a $10 million TAM investment whereas the latter required only $1 million. Benefit-cost (B/C) ratio: This measure allows comparisons across investments of different costs. The ratio is calculated as the present value of the benefits divided by the present value of the costs. A B/C ratio above 1.0 means that the benefits of a TAM investment outweigh the costs, whereas a B/C ratio below 1.0 means that the costs outweigh the benefits. The benefits and costs are exactly equal if the B/C ratio equals 1.0. B/C ratios can be used for investment prioritization, such that the investments with the highest ratios are made first. Internal rate of return (IRR): This measure is the discount rate at which benefits and costs are equal. IRR is expressed in percentage terms and can be considered to be the percent return that the investment provides. A project with an IRR greater than the discount rate used for the ROI analysis has benefits greater than costs and a positive economic value. IRR allows comparison of projects with different costs, different benefit flows, and different time periods. Payback period: This measure states the number of years it takes for the net benefits to equal (pay back) the initial investment costs. If a TAM investment is expected to pay back costs in 22 years but the initial investment will last only 10 years, the costs are never paid back. The payback period varies inversely with the B/C ratio (i.e., a shorter payback period yields a higher B/C ratio). For the ROI estimation of TAM investments, these four measures can be estimated from the perspective of the implementing agency or from the perspective of users and society. When estimating the ROI of TAM investments, agencies also may report other measures, such as total fuel cost savings, tons of greenhouse gases reduced, or full-time equivalent (FTE) savings in staff. For example, an investment in data collection automation may not actually result in fewer staff or a total reduction in agency costs, but it may result in more output (i.e., more assets being assessed). FTE equivalent may be one way to express this savings. Other examples are cost per output for geographical information systems (GIS) technology and data collection cost per mile for highway monitoring systems.12 Other potential metrics for estimating the impact of TAMS systems and practices relate to the change in asset conditions. The literature review found several examples, such as: • Remaining service life for pavement,13 • Pavement and bridge condition,14 and • Traffic-weighted average pavement condition by category.15 Consideration of Uncertainty ROI estimation is subject to considerable uncertainty. Whether the analysis is prospective or retrospective, at least one state of the world cannot be observed. The analysis therefore neces- sarily implies some form of prognostication or forecasting. The problem of uncertainty is com- pounded in the case of TAM systems for several reasons, including: • A relatively limited evidence base; • The complexity of cause-and-effect relationships between investments and effects; and • The multitude of internal and external factors that can affect outcomes.

Framework for Estimating ROI 21 Techniques are available to account for uncertainty in ROI analysis, and to assess or represent its impacts on the estimates of return.16 These techniques include sensitivity analysis, quantita- tive risk analysis, and other methods. Sensitivity Analysis In a sensitivity analysis, project-specific input values or model parameters are varied one at a time, and the resulting changes in project outcomes (e.g., NPV and B/C ratio) are reported accordingly. A sensitivity analysis may be used for multiple purposes, including: • Identifying the input values and model parameters whose variations have the greatest impact on the estimates of return (i.e., the so-called “critical” variables); • Evaluating the impact of changes in the critical variables on “reasonable” departures from their preferred, baseline values; and • Assessing the robustness of the results and, in particular, whether the general conclusions reached using the baseline assumptions are significantly altered through departures from those values. Figure 2-8 provides an example of an elasticity analysis, a type of sensitivity analysis developed through a series of tests applied to two input variables to determine how changes in the input values affect net benefits. The figure is for illustration only. The term sensitivity analysis also is occasionally used to describe instances when multiple assumptions are changed simultaneously. In the ROI estimation framework, this type of analysis is called scenario analysis. The limitations of sensitivity analysis have been documented in the literature. For exam- ple, Lewis (1995) argues that assumptions in sensitivity analysis often are varied by arbitrary amounts instead of by reference to reasoned analysis of potential error.17 In addition, varying Figure 2-8. Example of elasticity of net benefits with respect to changes in the value of travel time savings and the impact of pavement smoothness on fuel consumption—for illustration only.

22 Return on Investment in Transportation Asset Management Systems and Practices one assumption at a time does not provide an accurate view of the real world, where all factors affecting outcomes are likely to vary simultaneously. Nonetheless, this procedure can be useful for assessing the significance of individual assumptions in producing the overall ROI estimates. Sensitivity analysis has been used extensively in BCA and is recommended for BCA in support of some federal grant programs. Quantitative Risk Analysis Quantitative risk analysis measures the probability that an outcome (e.g., a B/C ratio greater than 1.0) will actually materialize. The measurement is accomplished by attaching ranges (or probability distributions) to the value of each input variable and model coefficient. This approach allows all inputs to be varied simultaneously within their distributions, thus avoid- ing the problems inherent in conventional sensitivity analysis. Quantitative risk analysis also recognizes correlations between variables and coefficients, and their associated probability distributions. Standard output from a quantitative risk analysis will include probability distributions for the variable of interest (in the form of histograms or cumulative probability distributions), as well as tornado diagrams, which are designed to help identify critical factors (i.e., those input variables that contribute most to the dispersion of simulated output). Tornado diagrams may be derived in multiple ways. Some simply illustrate the mean expected impact of risk vari- ables considered individually. Others are based on the change in output for a given change in input (typically one standard deviation) estimated through stepwise regression, or on the coefficient of correlation between input and output. Figure 2-9 shows a tornado diagram that ranks risk variables (threats and opportunities) by their expected impact value, in millions of dollars of net benefits. The figure is for illustration only as it is not based on any specific data or estimates. The probability distributions and correlation factors used in quantitative risk analysis typi- cally are derived from a mix of data, modeling, observations, and judgment. The risk analysis elicitation framework introduced by Lewis in 1995 illustrates how subjective probability can be Figure 2-9. Tornado diagram for key risks affecting net benefits—for illustration only.

Framework for Estimating ROI 23 used in the context of transportation planning and decision making.18 Risk analysis elicitation is defined broadly as a family of estimation techniques and planning processes used to examine risk and uncertainty, and to achieve public consensus through expert and stakeholder engagement. As defined by Lewis, the process involves four major steps: Step 1: Identification of the S&L of the estimation problem; Step 2: Assignment of estimates and probability distributions to each variable and coefficient in the S&L; Step 3: Expert and stakeholder engagement in the assessment of model and assumption risks; and Step 4: Issuance of risk analysis results. This framework has been used to provide decision support in several areas, including the quantification of airport and transit investment risk, traffic and revenue forecasting for toll roads, and the estimation of construction cost and schedule for large infrastructure projects. Comparable procedures have been developed in the energy, telecommunications, healthcare, and natural resources sectors. Because of its complexity and need for data with robust estimates of ranges of values for multiple variables, however, the risk analysis elicitation approach has not been incor- porated into the guidance provided in this report. The researchers found it unlikely that practitioners would currently have the data and resources available to apply this kind of analysis to TAM. Other Methods Other methods available to represent uncertainty in decision-support frameworks include scenario analysis and “what-if” analysis. Scenario Analysis In a scenario analysis, several assumptions (i.e., project-specific inputs or model parameters) are modified at the same time. A common form of scenario analysis consists of changing many or all assumptions in the same direction, once to produce an optimistic estimate and again to pro- duce a pessimistic estimate. The procedure is relatively easy to implement because the scenarios are simply defined as a collection of high (optimistic) and low (pessimistic) assumptions, without reference to the likelihood of these assumptions, taken either individually or jointly. Interpreting the outcomes of the scenario analysis also is generally straightforward and does not require any specific knowledge of probability theory. The concepts of high and low (or optimistic and pessimistic) are intuitive and generally understood. In most scenario analyses, the baseline results are presented as the most likely to occur, without any further analysis or assumption. The scenarios can be formed on the basis of data analysis or judgment. The use of high and low scenarios can be problematic, however, because it obscures the fact that the likelihood that all assumptions deviate from expectations in the same direction is extremely remote. In addition, as in what-if analyses, the lack of probability assessment limits the use and applicability of results. In estimating the ROI of TAM systems and practices, several assumptions can be subject to testing. Some examples include: • Asset conditions deteriorate at different rates than expected; • Treatments are not as effective as anticipated; • The assets experience more extreme weather or serve more users than expected;

24 Return on Investment in Transportation Asset Management Systems and Practices • Data quality is not substantially better than previously collected; and • Actual asset management investments reflect less than 100% of the recommendations by a TAM system. What-if Analysis This procedure, also called impact analysis, estimates the impact of a single event (such as an economic downturn or a rapid increase in fuel prices) and reports the results relative to a baseline estimate that is often presented as the “most likely” outcome. Uncertainties typically are assumed to be event specific, and are defined as either threats or opportunities. The procedure involves the following steps: Step 1: Establish a baseline estimate of return, assuming that none of the identified events will materialize; Step 2: Determine the magnitude of the events (e.g., severity and duration of an anticipated downturn in economic activity); Step 3: Using case studies from the literature (e.g., elasticity of travel demand with respect to fuel prices), or through judgment, determine the effect of each event, taken individu- ally, on the baseline estimate; and Step 4: Report the outcomes of the analysis. Impacts typically are reported one at a time, with no reference to potential dependencies or correlations with other events. One of the main strengths of the what-if approach is that it is relatively simple to implement and describe to non-technical audiences. On the other hand, the use of this approach requires reference to a central, most likely outcome, and the probability of alternative outcomes under different conditions. Given that these probabilities are typically unknown, the interpretation of results can be somewhat difficult. Chapter Endnotes 1. U.S.DOT, Tiger Benefit-Cost Analysis (BCA) Resource Guide, April 2, 2015, and U.S.DOT, TIGER BCA Guidance, April 6, 2015. 2. Benefit-Cost Analysis Guidance for TIGER Grant Applicants, found at https://cms.dot.gov/sites/dot.gov/ files/docs/TIGER_BCA_Guidance.pdf, page 4. 3. If one accepts the idea that BCA is a comparative method, then it is easy to understand why an unrealistic base case would lead to unrealistic results. 4. U.S. GAO, Highway and Transit Investments: Options for Improving Information on Projects’ Benefits and Costs and Increasing Accountability for Results, GAO-05-172, 2005, page 14. 5. See FHWA, Economic Analysis Primer, Washington, D.C., August 2003, available at www.webpages.uidaho. edu/∼mlowry/Teaching/EngineeringEconomy/Supplemental/USDOT_Economic_Analysis_Primer. pdf and Bureau of Transport Economics, Facts and Furphies in Benefit-Cost Analysis: Transport, Research Report 100, November 1999, available at www.bitre.gov.au/publications/1999/report_100.aspx. Note: Fur- phies (plural of furphy) is an Australian slang term that describes erroneous stories or information that are claimed as factual. 6. See Bernhardt, S. and S. McNeil, Impacts of Condition Assessment Variability on Life Cycle Costs, American Society of Civil Engineers, 2006; Ye, Z., C. Strong, X. Shi, S. Conger, and D. Huft, Benefit–Cost Analysis of Maintenance Decision Support System, in Transportation Research Record: Journal of the Transportation Research Board, No. 2107, Transportation Research Board of the National Academies, 2009; and McNeil, S., D. Mizusawa, S. Rahimian, and J. Bittner, Assessing and Interpreting the Benefits Derived from Implementing and Using Asset Management Systems, Project 06-06, Phase 2, Midwest Regional University Transportation Center, June 2011. 7. See De la Garza, J. M., C. G. Howerton, and D. Sideris, A Study of Implementation of IP-S2 Mobile Map- ping Technology for Highway Asset Condition Assessment, Virginia Tech, 2011; Nokes, W., L. du Plessis, M. Mahdavi, N. Burmas, T. J. Holland, and J. Harvey, Tools and Case Studies for Evaluating Benefits

Framework for Estimating ROI 25 of Pavement Research, 8th International Conference on Managing Pavement Assets, 2011; Schiffer, A., Automated Asset Inventory System, Arizona Department of Transportation, April 2006; and Yen, K. S., B. Ravani, and T. A. Lasky, LiDAR for Data Efficiency, Washington State Department of Transportation, September 2011. 8. See FHWA, An Analysis of the Economic and Non-Economic Costs and Benefits of Implementing MAP-21 Asset Management Plans and Related Provisions, 2015, and Spy Pond Partners, NCHRP Report 800: Suc- cessful Practices in GIS-Based Asset Management, Transportation Research Board of the National Academies, Washington, D.C., 2015. 9. Mishan, E. J., and E. Quah, Cost-Benefit Analysis, 5th ed., Routledge, 2007. 10. de Neufville, R., Applied Systems Analysis: Engineering Planning and Technology Management, McGraw Hill, 1990. 11. U.S. Office of Management and Budget, Regulatory Analysis, Circular No. A-4, September 17, 2003, available at www.whitehouse.gov/sites/default/files/omb/assets/omb/circulars/a004/a-4.pdf. 12. See Martland, C., Toward More Sustainable Infrastructure: Project Evaluation for Planners and Engineers. John Wiley & Sons, 2012; Dye Management Group, Inc., Monitoring Highway Assets with Remote Technol- ogy, Michigan Department of Transportation, July 2014. 13. Hendren, P., Transportation Research Circular E-C076: Asset Management in Planning and Operations: A Peer Exchange, Transportation Research Board of the National Academies, Washington, D.C., 2005. 14. Ibid. 15. McNeil, S., and D. Mizusawa, Measuring the Benefits of Implementing Asset Management Systems and Tools, Midwest Regional University Transportation Center, University of Wisconsin, Madison, 2008. 16. Additional information and guidance on these and other techniques can be found in Granger, M., and M. Henrion, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press, 1990, or in Kincaid, I., et al., ACRP Report 76: Addressing Uncertainty about Future Airport Activity Levels in Airport Decision Making, Transportation Research Board of the National Academies, Washington D.C., 2012. 17. Lewis, D., “The Future of Forecasting: Risk Analysis as a Philosophy of Transportation Planning,” TR News 177, March-April 1995, Transportation Research Board of the National Academies, Washington, D.C. 18. Ibid.