Return on Investment in Transportation Asset Management Systems and Practices (2018)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

145 A P P E N D I X E Description of the Southern State Case Study This appendix provides additional details on the Southern State case study described in Chapter 3. The case study was performed to determine benefits realized through the DOT’s implementation of a maintenance levels of service approach and new TAM system. The following subsections supplement the materials in Chapter 3 with additional information on the analysis approach and results. ANALYSIS APPROACH The case study analysis included the following basic steps: First, data were collected on maintenance elements, LOS ratings, annual weather data, and construction cost inflation. Second, the data were reviewed to ensure there were no duplicates, outliers, or gaps. The maintenance LOS data were pooled at the category level so they could be matched to categorical spending. Third, the research team applied descriptive statistics, data visualization, and simple regression models to understand the trends in the data. Fourth, a statistical model was developed using multiple linear regression to understand the change in scores over time. Multiple models and specifications were tested using the data. Dependent variables in the model included LOS ratings by asset category, district and year. Independent variables included spending by category, district and year, as well as annual weather statistics (mean temperature). Fifth, regression diagnostics were run after fitting the model to evaluate the model assumptions and to investigate if any observations largely influenced the analysis. Sixth, the research team examined the model to interpret the results and found that an interaction variable showed that the implementation of the TAM improved the cost- effectiveness of M&R spending. These steps are detailed in the following subsections. Data Collection The DOT conducts yearly inspections to assess the conditions of up to 80 maintenance elements. These elements are grouped into maintenance categories such as pavement, bridge, roadside, and traffic services. The maintenance elements describe potential asset failures or essential structures that need to be monitored to ensure the preservation and safety of the highway system. Examples of maintenance elements include potholes, rutting, shoulders, traffic signs, and cross drains. The research team pulled LOS scores from the DOT’s annual maintenance summary reports for fiscal years 2008 to 2014. Although LOS data were also available for 2007, data from this year were

146 Return on Investment in Transportation Asset Management Systems and Practices collected using a slightly different approach. The reports provided annual data on LOS letter grades by district for each maintenance element, split by category for all districts. Data were available for all six of the DOT’s districts. Table E.1 displays the potential LOS letter grades and their corresponding numeric scores. Table E.1. LOS Letter Grades and Scores. Letter Grade Score A+ 14 A 13 A- 12 B+ 11 B 10 B- 9 C+ 8 C 7 C- 6 D+ 5 D 4 D- 3 F+ 2 F 1 F- 0 Each maintenance report also includes information on that year’s LOS objectives for each maintenance element. The objectives are fairly consistent across districts and years except for a change in objectives that occurred in 2013. The early reports (2007 to 2009) do not provide LOS objectives, so the research team chose to assume that the 2010 objectives apply to these earlier years. The maintenance reports also provide annual spending for each maintenance function by maintenance type as well as summaries of the roadway inventory (e.g., annual road and lane miles). To help control for the effects of weather on maintenance condition, the research team collected annual weather data from the National Oceanic and Atmospheric Agency (NOAA) National Climatic Data Center. The research team also collected data from the Engineering News Record (ENR) annual construction cost indices (CCI) to inflate the maintenance cost data from earlier years into 2014 dollars. Data Cleaning and Analysis After reviewing and cleaning the available data for formatting problems or missing data, the research team created a database for further analysis. The following transformations were made to create this database:

Appendix E 147 Mapped LOS maintenance categories to maintenance expenditure types and ensured consistent naming conventions over the years; Adjusted maintenance spending data for inflation; Estimated lagged maintenance costs; Aggregated total spending by maintenance category; Pooled LOS data at the category level; and, Mapped districts to NOAA weather stations and average temperature. The research team studied the annual scores for each maintenance element and district. Figure E-1 shows the categorical averages of the LOS scores. Each dot represents a maintenance category and one of the DOT’s six districts with some dots lying on top of each other. As can be seen in the figure, the LOS scores for individual maintenance categories cover the full range of potential score from 0 (F-) to 14 (A+). The red line illustrates trend in average scores from 2008 to 2014. Overall, the trend in scores is downward, indicating a worsening of conditions. This initial examination of the data suggests that implementation of the new TAM system in 2010 did not improve the average LOS scores. Figure E-1. LOS Scores for Each Maintenance Category and District (2008–2014). However, the LOS scores are affected by a number of other factors, such as maintenance spending and weather over the same time period. Figure E-2 shows the total maintenance costs per mile in 2014 dollars from 2008 to 2014. Since the effects of maintenance activities are not likely to be reflected in LOS scores until the following year, the maintenance costs are lagged by one year (e.g., the data for 2008 corresponds to the maintenance costs in 2007). The maintenance costs per mile have varied considerably from 2008 to 2014, but one trend is readily apparent. The maintenance expenditures per mile dropped at the same time that the new TAM system was implemented. As a result, it is difficult to separate the impacts of the TAM system and maintenance expenditures on the LOS scores.

148 Return on Investment in Transportation Asset Management Systems and Practices Figure E-2. Total Lagged Maintenance Costs per Mile (in 2014 dollars, 2008–2014). After further exploratory analysis, the research team found that the difference in LOS scores from the corresponding LOS objectives for the year provided more significant results than the raw scores alone. This finding makes sense, as it is likely an agency would be more interested in spending to meet its objectives rather than to maximize its LOS scores. Table E-2 shows average LOS scores and the difference in scores from LOS objectives broken down by district. On average, the scores of District 6 have the best performance as they surpass the objectives by 1.5 grade points, while the scores of District 3 have the weakest overall performance as they only surpass the objectives by 0.23 grade points. Table E-2. Average LOS Scores by District. District Average Difference in Scores (Score-Objective) Median Difference in Scores (Score-Objective) Average LOS Scores Median LOS Scores 1 0.688 1.0 7.733 9.0 2 0.716 1.0 7.731 8.0 3 0.262 0.5 7.300 8.0 4 0.678 1.0 7.716 8.0 5 1.209 2.0 8.181 9.0 6 1.525 2.0 8.562 10.0 Table E-3 summarizes the LOS score by maintenance category. The average scores of the Drainage and Traffic Services maintenance categories fail to meet the objectives by 2.194 and 0.07 grade points, respectively. The roadside maintenance categories have the highest average scores, surpassing the objectives by 2.276 grade points.

Appendix E 149 Table E-3. Average LOS Score by Category. Category Average Difference in Scores (Score-Objective) Median Difference in Scores (Score-Objective) Average LOS Scores Median LOS Scores Asphalt Roadway 0.853 1.0 7.381 8.0 Bridge 1.528 2.0 10.528 11.0 Concrete Roadway 1.451 2.0 7.980 9.0 Drainage -2.194 -5.0 3.806 1.0 Rest Areas and Welcome Center 2.000 1.5 11.000 10.5 Roadside 2.276 2.0 7.027 7.0 Shoulder 1.671 1.0 8.515 9.0 Traffic Services -0.070 1.0 8.987 11.0 The LOS scores and their differences from the LOS objectives were compared against three variables available in the database (e.g., time, lagged spending per mile, and temperature). The lagged spending per mile variable was used because the research team expected that any maintenance spending would improve the LOS conditions in the following year. Spending was divided by road mile for each district so that all districts could be weighted equally.

150 Return on Investment in Transportation Asset Management Systems and Practices The research team found the following relationships through the initial analysis: The average LOS scores have been declining over time. When spending increases, the actual scores surpass the LOS objectives. The LOS objectives are harder to attain in districts and years with colder temperatures. Total maintenance spending per mile has declined on average by 13.7 percent annually since the TAM system was implemented. On average, weather conditions were worse after TAM system implementation. Statistical Model The research team tested several different statistical models to fit the available data. Correlations among the explanatory variables were studied to ensure that there would not be multi-collinearity when running multiple regressions, which is a statistical condition that would make the results difficult to interpret. The research team found that none of the predictor variables were highly correlated with one another In the end, the research team fit the following multiple linear regression model using ordinary least squares (OLS): ln . Where, Reflects the difference in the maintenance scores and the LOS objectives for the element in the district in the year and is the number of elements in the category. Essentially, this equation estimates the likely deviation in maintenance scores from the LOS objectives by taking into account: The presence of the new TAM system (TAM); The amount of maintenance spending per mile; The interaction between the new TAM system and maintenance spending per mile; Unique maintenance needs or conditions present in each district (District);

Appendix E 151 LOS scores by maintenance category, district and year (CTGY); and, Average temperatures in each district by year, represented by the departure from normal monthly mean temperature (DPNT). Table E-4 shows the coefficients of the model estimated for the case study. Many of the terms are statistically significant, including the TAM system and maintenance spending interaction variable – a key finding for the study. The adjusted is approximately 20 percent. This shows there is a correlation between this factor and the resulting LOS scores, but that the model explains only a portion of the observed variation. Other factors that contribute to the variation observed in LOS values may include impact of capital investments not included in maintenance spending, lags in condition as a result of historic spending (e.g., conditions may still decline in a period of increased spending if there is a backlog of needs), inherent subjectivity in human observations of LOS, and inherent uncertainty introduced by use of a sampling approach to collect LOS data. The research team tested several other forms of the model, but this one was chosen for its good fit, outcome, and simplicity. Table E-4. Coefficients of the Multiple Linear Regression Model. Model Unstandardized Standardized -value2 2.488 0.818 10.528 3.042 0.002 ** -4.736 0.681 -0.544 -6.958 0.000 *** 0.053 0.106 0.020 0.502 0.616 0.498 0.107 0.373 4.661 0.000 *** -0.180 0.292 -0.015 -0.614 0.539 -0.470 0.287 -0.041 -1.638 0.102 -0.240 0.289 -0.021 -0.830 0.406 0.447 0.292 0.038 1.531 0.126 0.454 0.300 0.039 1.513 0.130 1.408 0.337 0.114 4.178 0.000 *** 1.800 0.530 0.135 3.397 0.001 *** -2.790 0.327 -0.208 -8.542 0.000 *** 0.911 0.595 0.031 1.530 0.126 1.222 0.288 0.108 4.249 0.000 *** 1.129 0.372 0.070 3.039 0.002 ** -0.951 0.282 -0.084 -3.368 0.001 *** -0.224 0.087 -0.052 -2.583 0.010 ** -1.675 0.171 -0.194 -9.766 0.000 *** 2 Significant codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ‘ 1

152 Return on Investment in Transportation Asset Management Systems and Practices Model Diagnostics The research team ran regression diagnostics after fitting the model to evaluate assumptions and to investigate if any observations were largely influencing the analysis. Model diagnostic procedures involve both graphical methods and formal statistical tests. The four main assumptions for constructing a regression model are linearity, homoscedasticity, independence, and normality. The research team tested each of these assumptions. The relationship between the dependent variable and the explanatory variables must be linear. The errors are independent and identically distributed (i.i.d.) values which follow a normal distribution with a mean of 0 and constant variance . The Durbin-Watson statistic was used to detect the presence of serial correlation among the residuals. The value of the Durbin-Watson statistic ranges from 0 (strong positive correlation) to 4 (strong negative correlation). A value of 2 generally indicates that the residuals are uncorrelated. In the case study regression, the value of the Durbin-Watson is approximately equal to 2, indicating no serial correlation. The adjusted determines how well data fit the model. The case study regression shows a low value of approximately 0.2 or 20 percent because there are a fixed set of fitted Y values. There is only a finite set of outcomes based on the limited range of the explanatory variables since most of them are indicator variables. As a result, the model should not be used to predict reasonably precise scores. Since the scores vary per year per district per element, the model can never explain all the variance across the many different elements per year per district. Figure E-3 displays four diagnostic plots. The first plot depicts the residuals against the fitted values, which can be used to study whether the regression function is nonlinear and the error terms have non-constant variance. There is no evidence that the model does not meet the linear model assumption, because the residuals do not show any very large positive or negative values. As there is no visible pattern in the residuals and they appear to be randomly and equally scattered about 0, the research team found that the constant variance assumption holds.

Appendix E 153 Figure E-3. Diagnostic Plots of Residuals. The second plot displays a histogram of the model’s residuals. The black curve represents the normal density function with mean 0 and the standard deviation of the residuals. The observations closely follow the normal distribution. The normal probability plot of the residuals, which checks the normality assumption, is shown in the third plot. Here each residual is plotted against “ideal” normally distributed observations. The observations lie well along the 45 degree line, implying that the error terms are normally distributed. The final plot is of “Cook’s distance,” which is a commonly used measure that determines whether or not a point is influential on the regression coefficients. An influential point is one if removed from the data would significantly change the fit. It could be either an outlier or have large leverage. Typically, points with Cook’s distance greater than 1 are classified as being influential. As Cook’s distance is close to 0 for all observations, then no observation has a strong influence on the fit of the statistical model. Model Interpretation and Analysis The research team made the following observations about the coefficients in the regression model: Interaction effect of cost and TAM period o If all other variables are fixed and there is no change in spending, then on average the scores are worse than the objectives by 4.7 grade points after the TAM system is implemented.

154 Return on Investment in Transportation Asset Management Systems and Practices o The effect of the cost of the mean departure from the objectives is not statistically significant before the TAM system is implemented. o The interaction of cost per mile and the TAM system is a significant positive predictor. After TAM system implementation, for every 10 percent increase in cost per mile, the mean scores surpass the objectives by approximately 0.05 units. Categories: o The average response for all categories is significant relative to the asphalt category, except for rest areas. The drainage and traffic services categories have the lowest average outcomes. Temperature: o If all other variables are fixed, then for each 10 percent change in the average annual departure from the normal temperature, the average scores are approximately 0.02 grade points worse than the objectives. o The average scores are 1.7 grade points worse than the objectives when the change in temperature is zero or negative. RESULTS Summary Results The critical finding from the statistical model is the interaction between the new TAM system and the efficiency of maintenance expenditures ( . Although performance levels have declined after the new TAM system was implemented, the cost effectiveness of money spent on maintenance has improved. The interaction of maintenance expenditures per mile and the presence of the TAM system is positive and statistically significant. According to the regression model results, for every 10 percent increase in cost/mile, the mean scores surpass LOS objectives by approximately 0.05 units after TAM. The model shows that during post-TAM, it is more efficient to maintain an objective for every dollar spent. In the pre- TAM period, a considerable amount of money was spent without positively affecting the scores. A possible interpretation of that outcome is that without a sophisticated maintenance management system during the pre-TAM period, the budget was not being effectively distributed across the districts and maintenance elements. Essentially, prior to the new TAM system, money was being spent on maintenance elements that were already scoring high on the LOS scale. The regression model estimated in this case study can be used to assess the impact of TAM implementation, as detailed further below. However, it is not possible to assess the impact for this case study as the model for pre-TAM implementation fails the test of statistical significance. However, further detail on how to use the model results to demonstrate implementation impact is provided below for use in any future case studies for which sufficient data are available to support a times series analysis such as that described here. Assessing the Efficiency of a TAM System The research team developed equations to compare the percent change in maintenance expenditures per mile and the expected change in scores. If is the percent change in expenditures per mile and is the expected change in scores,

Appendix E 155 then the overall expected change in scores ( ) given the percent change in the lagged expenditure per mile ( ) is: where TAM indicates whether the expenditures are pre-TAM system (0) or post-TAM system (1). Conversely, the percent change in the expenditure per mile ( ) given the expected change in scores ( ) can be calculated as: If coefficients on expenditures b per mile are significant in both the pre- and post-TAM periods, then the efficiency of the TAM system can be estimated as follows: ln ln ln . Thus, given the same lagged spending per mile, if the result above is positive, then the improvement in scores after implementation of the TAM system is grade points more. The amount less that needs to be spent post-TAM compared to pre-TAM to achieve the same outcome in scores can be estimated as: . Application to the Case Study Since the regression equation shows that spending is statistically significant post-TAM implementation, the formulas above can be applied to determine the spending needed to improve LOS scores, but they cannot be used to obtain meaningful results pre-TAM implementation because the change in scores as a function of expenditures was not statistically significant. The savings in maintenance expenditures per mile after investing in the TAM system could be estimated from the regression equation if the model had a high adjusted ( ) and a variable that is statistically significant. Unfortunately, these conditions are not met for the case study, but may have been had more data been available. In the event sufficient data were available to develop a statistically significant model, one could estimate how much would need to be spent per mile in each period to meet the LOS objectives given that all other factors are equal. The savings from TAM investment across all districts and categories could then be expressed: where, ln

156 Return on Investment in Transportation Asset Management Systems and Practices Although it was not possible to demonstrate the effect of implementing the TAM system for the case study, the available data showed significant annual increases in average annual cost per mile from fiscal years 2007 to 2009 followed by significant drops in cost per mile from fiscal years 2010 to 2012. Afterwards, annual average cost per mile started to increase. Also, the temperature (DPNT) dropped significantly after fiscal year 2010 which indicated a period of colder than average monthly temperatures (average annual DPNT during the pre-TAM period was 0.72 and the average during the post-TAM period was -1.2). Coincident to the drop in annual expenditures and the DPNT, the performance of LOS scores relative to objectives also dropped significantly. The modeling exercise identified a statistically significant relationship between increased spending and score improvements relative to objectives (or conversely, decreased spending and reduced score performance relative to objectives). This finding demonstrates the efficiencies gained by implementing the TAM system. However, in order to use the model’s estimates to measure the average savings in maintenance expenditures, a longer time series of data should be collected during the pre-TAM period and during the period when the TAM system was fully operational. Further, the costs of staff time for implementing the system are not included in the analysis, and including these costs would lower the predicted net benefits of system implementation. The following would likely have been required to better assess the effects of TAM system implementation in the case study: A minimum of five years of data before and after TAM system implementation. Each maintenance element should be associated to a specific cost rather than to the cost at the category level. The definitions of LOS scores for maintenance elements should be consistent from year to the next. Vehicle-miles traveled data should be collected and used as a predictor in the model.