Improving Motor Carrier Safety Measurement (2017) / Chapter Skim
Currently Skimming:
2 Overview, Evaluations, and Criticisms of the Safety Measurement System
2 Overview, Evaluations, and Criticisms of the Safety Measurement System
Pages 23-62
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 23... ...
Such a prediction would take into consideration multiple factors, such as the length and nature of the carrier's routes, schedule, cargo, and driver characteristics. Unfortunately, such a program is not feasible, in part, because in any given time period, the incidence of observed crashes is small even though the risk of a crash for a carrier might still be high, and because FMCSA does not have access to motor carrier, vehicle, and driver data that could be used to calculate crash risk.
|
From page 24... ...
However, if the objective is that of prevention, there does not need to be a direct causal link between the frequency of occurrence of many of the violations and future crash risk, since their productive use in SMS only depends on the assumptions that carriers that are frequent violators engage in unsafe practices, and carriers that engage in unsafe practices are also likely to have a high frequency of future crashes. So, for example, while a truck or bus often cited for "minor lights out" may not as a direct result be involved in crashes, carriers that are not meticulous about such things may have more crashes due to a general poor approach to vehicle maintenance that impacts crash risk.
|
From page 25... ...
Later in this chapter, we describe FMCSA's evaluations of the degree to which the assumption of the linkage between the frequency of inspection violations and future crash risk obtains. The data used in SMS are those primarily collected during inspections by Motor Carrier Safety Assistance Program (MCSAP)
|
From page 26... ...
, but also the measures of carriers have, until recently, been made public so that the motor carrier industry and other safety stakeholders would have access to comprehensive and regularly updated safety performance data. In addition, the hope is that by doing this, motor carriers will have an incentive to improve their SMS measures (relative to their peers)
|
From page 27... ...
The exception is that for the Unsafe Driving BASIC, the denominator is essentially an estimate of vehicle miles traveled, which is arrived at by multiplying the number of power units a carrier has by a utilization factor (see Chapter 6 for more detail)
|
From page 28... ...
For each BASIC, and for each combination and straight segment, FMCSA has derived thresholds for use with the associated percentile ranks that are the same across safety event groups, which comes to a
|
From page 29... ...
This BASIC uses crash history that is not specifically a behavior but instead the consequence of a behavior or a set of behaviors." The measure used is Total of time and severity weighted violations Crash Indicator Measure = Total time weight of relevant inspection SOURCE: Federal Motor Carrier Safety Administration (2016a)
|
From page 30... ...
Further, Table 2-2 shows that separately for each BASIC, the crash rate for carriers with alerts is considerably higher than the national aver 3 Carrier Safety Measurement System (CSMS) Effectiveness Tests by Behavior Analysis and Safety Improvement Categories (BASICs)
|
From page 31... ...
In such an analysis, the comparison would not be the national statistic, it would be the statistic for the remaining carriers in the safety event group. A surprising result is that carriers with alert status in the Driver Fitness BASIC have lower crash rates than the national average.
|
From page 32... ...
is to examine, across all safety event groups, the number of future crashes for the carriers that receive such notifications under each system with the algorithm hav ing the greatest number of future crashes the winner. There would be some car riers that are identified by both methods and some by neither method, and there would be carriers identified by SMS but not by SMS*
|
From page 33... ...
SOURCE: Carrier Safety Measurement System Effectiveness Test by Behavior Analysis and Safety Improvement Categories (January 2014)
|
From page 34... ...
NOTES: BASIC, Behavior Analysis and Safety Improvement Categories; HM, hazardous materials; HOS, hours of service. SOURCE: Compliance, Safety, Accountability Effectiveness Measures (June 30, 2016)
|
From page 35... ...
/7.64 5 |
From page 36... ...
SMS, Safety Measurement System. SOURCE: Compliance, Safety, Accountability Effectiveness Measures (June 30, 2016)
|
From page 37... ...
DISCUSSION OF CONCERNS RAISED ABOUT SMS General View of the Approach Taken in SMS Given the assumption that carriers that violate safety provisions more frequently are also those that have a higher future crash risk, it is reasonable to use the percentage of (weighted) inspections that have violations that are associated with a specific type of safety deficiency, as metrics for each carrier.
|
From page 38... ...
This is important since the intention is to make SMS percentile ranks public. While there are no studies that we are aware of that have attempted to measure the size of the economic impact from making SMS ranks public, it is reasonable to expect that the impact would be substantial.
|
From page 39... ...
ATRI (b) showed substantial state differences in crash rates, which suggests that some states are riskier to drive in, which should be taken into account when forming the denominators for Crash Indicator and possibly for Unsafe Driving.
|
From page 40... ...
ATRI (a) tried to validate severity weights through Weights use of a logistic regression model of crash risk as a function of data on violations, crashes, and carrier characteristics.
|
From page 41... ...
ATRI, American Transportation Research Institute; BASICs, Behavior Analysis and Safety Improvement Categories; FMCSA, Federal Motor Carrier Safety Administration; GAO, U.S. Government Accountability Office; MCMIS, Motor Carrier Management Information System; SMS, Safety Measurement System.
|
From page 42... ...
have shown some of the BASICs, especially Crash Indicator and Unsafe Driving, have very strong correlations with future crash risk, and three more BASICs have moderately strong correlations. However, Driver Fitness has been shown to have a negative correlation with future crash frequency.
|
From page 43... ...
Again we see TABLE 2-11 Relative Crash Risk for Carriers Above versus Below Alert Threshold in Each BASIC BASIC Crash Risk Unsafe Driving 1.74 Vehicle Maintenance 1.42 HOS Compliance 1.34 Controlled Substance/Alcohol 1.32 Driver Fitness 0.87 NOTE: BASIC, Behavior Analysis and Safety Improvement Catego ries; HOS, hours of service. SOURCE: American Transportation Research Institute (2012)
|
From page 44... ...
, a BASIC percentile rank in one category that is not strongly correlated with future crash risk may still be predictive when combined with other BASIC percentile ranks. For example, regressing crash rate on BASIC percentile ranks with interaction terms can reveal underlying relationships between the BASICs.
|
From page 45... ...
TABLE 2-13 Crash Rates for Carriers Identified by the Safety Measurement System Compared to Those Not Identified Crash Rate per 100 Ratio to Not BASIC Threshold Exceeded Carriers Crashes Power Units Power Units Identified Unsafe Driving 9,245 33,532 450,874 7.44 3.56 Fatigued Driving 17,959 15,525 248,862 6.24 2.99 Driver Fitness 3,981 11,539 379,009 3.04 1.46 Controlled Substance and 1,013 6,860 104,799 6.55 3.14 Alcohol Vehicle Maintenance 18,280 13,643 278,198 4.90 2.34 Improper Loading/Cargo 9,409 16,747 421,670 3.97 1.90 Securement Crash Indicator 5,077 33,946 463,766 7.32 3.51 SOURCE: Green and Blower (2011)
|
From page 46... ...
Data Sufficiency Standards Data sufficiency standards must trade off the reliability of SMS measures and percentile ranks with the percentage coverage of the carrier population that is given SMS measures. As data sufficiency standards are relaxed, resulting in less reliable SMS measures, it is possible to provide SMS percentile ranks for a larger fraction of the active CMV carriers.
|
From page 47... ...
9) recommended that FMCSA: "Continue to identify and implement methods for emphasizing absolute rather than relative individual motor carrier rankings so that it does not undermine industry's willingness to innovate and share best practices." The team based its recommendation on the following conclusion: The relative SMS percentile ranks motor carriers based on their SMS scores relative to their peers.
|
From page 48... ...
RECOMMENDATION: Given that there are good reasons for both an absolute and a relative metric on safety performance, Federal Motor Carrier Safety Administration should decide on the carriers that receive Safety Measurement System (SMS) alerts using both the SMS percentile ranks and the SMS measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
|
From page 49... ...
In a 2012 FMCSA report (Craft, 2012) , researchers using police accident report data coded 1,221 crash records across five severity categories, with 93.2 percent agreement with assessments provided by researchers using data from the large truck crash causation study (Blower and Campbell, 2002)
|
From page 50... ...
We believe that such research is of interest. While we are skeptical about setting such data aside, there might very well be schemes in which downweight crashes judged to be nonpreventable -- even if the method for arriving at such a determination is error-prone -- could result in an SMS percentile rank for Crash Indicator that is preferable to the current version.
|
From page 51... ...
. ." Further, ATRI showed that if these state differences were eliminated, SMS percentile ranks would change appreciably.
|
From page 52... ...
Stratification of Types of Carriers In order to provide for a fair comparison of carriers, SMS would benefit from stratification that formed peer groups of carriers that were undertaking trips of similar risks. This is somewhat taken care of by standardizing by vehicle miles traveled for the Crash Indicator BASIC, but, as discussed above, trips through some states and some roads at some times of the year and at some times of the day are riskier than others.
|
From page 53... ...
On the other hand, further stratification results in having fewer peers, and the fewer peers that a carrier has, the more difficult it is to be certain that a carrier's performance is atypical by being among the highest by a certain percentage. Trading off these two considerations is difficult, assuming the desire to retain the same number of safety event groups throughout the stratification.
|
From page 54... ...
APU also has some nonresponse, though much less than for VMT. Given that this factor can clearly be off by a substantial amount and has a direct impact on the Crash Indicator BASIC and the Unsafe Driving BASIC, it is vitally important for FMCSA and CMV associations to work collaboratively on an improvement.
|
From page 55... ...
In addition, such improvements will not only improve the Crash Indicator BASIC as an indicator of which carriers receive interventions, but also as an improved measure of future crash risk. Appropriateness of Severity Weights ATRI (2012)
|
From page 56... ...
We do not believe that research into the relationship between severity weights and future crash risk should be a high priority for FMCSA since the algorithm is not extremely dependent on such weights. Also, such research is similar to the models that were argued to be too difficult to development earlier in this report.
|
From page 57... ...
There are currently efforts to mandate that these software packages be standardized to some extent to map the observations of inspectors to violations in the same way. Clearly, the fact that identical situations can result in very different violation codes with different severity weights and the lack of any national standardization of the coding that is done adds additional variability to the percentile ranks, which makes it more difficult to identify the carriers operating less safely than others.
|
From page 58... ...
One measure of agreement is the percentage of those with alert status for a given BASIC also getting alert status for the BASIC that is being compared. This percentage is as high as 55 percent for Driver Fitness compared to Vehicle Maintenance, and 34 percent for Hours of Service and Unsafe Driving.
|
From page 59... ...
But such carriers are peers. Their performance as peers is relevant to understanding what is feasible, and to the relationship between SMS percentile ranks and future crash frequency, and therefore the information is critical to know.
|
From page 60... ...
The variables that define safety event groups are the number of relevant inspections for five BASICs, the number of crashes for the Crash Indicator BASIC, and the number of inspections with an unsafe driving violation for the Unsafe Driving BASIC. However, the difference between the current definition of safety event groups and size is fairly modest, so our analysis proceeded as if the categories were defined by size.
|
From page 61... ...
We are aware of one other problem raised by safety event groups. We were told by the representative of a carrier that between 2 months, its BASICs did not appreciably change but its percentile ranks increased substantially.
|
From page 62... ...
This argues for Federal Motor Carrier Safety Administration adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.