A Review of the Department of Transportation's Plan for Analyzing and Testing Electronically Controlled Pneumatic Brakes Letter Report (Phase 2) (2017)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

  1 LETTER REPORT ON A REVIEW OF THE DEPARTMENT OF TRANSPORTATION TESTING AND ANALYSIS RESULTS FOR ELECTRONICALLY CONTROLLED PNEUMATIC BRAKES INTRODUCTION The U.S. Department of Transportation’s (DOT’s) final rule entitled Enhanced Tank Car Standards and Operational Controls for High-Hazard Flammable Trains [HHFT] includes requirements designed to reduce the consequences and, in some cases, the probability of accidents involving trains transporting large quantities of flammable liquids, including crude oil and ethanol.1,2 The HHFT rule, on the basis of the estimated effectiveness of electronically controlled pneumatic (ECP) brakes in diminishing the conse- quences of derailment events by more rapidly reducing the kinetic energy of a train during a derailment, requires the use of ECP brakes on unit trains designated as high-hazard flammable unit trains (HHFUTs). According to the rule, an HHFUT is a train comprising 70 or more loaded tank cars transporting flamma- ble liquids at speeds in excess of 30 mph. HHFUTs are required to use ECP brakes in the transport of tank cars loaded with flammable liquids with specified characteristics and traveling at speeds greater than 30 mph by January 2021. During development of the rule, DOT considered the potential effectiveness of pneumatic (air) brakes and three enhanced pneumatic braking systems in reducing the number of tank cars derailed or punctured in the event of a train accident. Pneumatic brakes (also referred to as conventional brakes) in- volve the use of air compressors on locomotives to charge the train car brakes on each of the coupled cars making up a train, which is also referred to as a consist. When the train’s engineer applies the brakes, a brake pipe pressure reduction is initiated at the locomotive, which creates a pressure wave in the brake pipe (or train line) that propagates down the length of the train. When the pressure reduction wave reaches a car, a control valve on that car activates the brake cylinder, which causes the brake shoes to apply to the wheels. The brake pipe air pressure reduction also brings about the activation of brakes in the next car. The process is repeated sequentially until the air pressure reduction reaches the end of the train. The braking systems considered in the rulemaking involved the use of pneumatic brakes with a dis- tributed power (DP) configurations, pneumatic brakes with two-way end-of-train (EOT) devices, and ECP brakes. EOT devices and DP configurations are commonly used by railroads in the United States in con- junction with pneumatic brakes. All single-commodity trains (usually referred to as “unit trains”) in the United States carrying crude oil or ethanol use either EOT devices or DP configurations to enhance their pneumatic brake systems, and the HHFT final rule requires that either system be used on HHFTs. DP configurations use locomotives positioned at strategic locations within the train consist (usually at the rear of the train, but sometimes also in the middle) to provide additional power and train control in certain operations (such as climbing steep inclines). For emergency braking involving DP consists with a locomotive at the rear, brake application can be initiated in both directions from the front and the rear of the train via radio transmission. For emergency braking involving EOT systems, when emergency conditions (such as brake pipe separation) are detected near the front of the train, a device at the front of the train sends a radio signal to the EOT at the rear of the train to activate emergency brake application from the rear of the train. If an emergency condition (such as a brake pipe separation) occurs in the rear half of the train, the system is programmed so that when the EOT detects a loss of brake pipe pressure, it will send a radio signal to the device at the front of the train. The engineer then decides when to initiate an emergency brake application on the basis of circumstances ahead (such as disturbed track). 1The final rule was issued by DOT’s Pipeline and Hazardous Materials Safety Administration and the Federal Railroad Administration at 80 Federal Register 26644–26750 (May 8, 2015). 2The rule defines an HHFT as “a train comprised of 20 or more loaded tank cars of a Class 3 flammable liquid in a continuous block or 35 or more loaded tank cars of a Class 3 flammable liquid across the entire train.”

2 According to DOT, ECP brake systems reduce the time before a unit train’s pneumatic brakes are fully engaged and therefore are more effective than other pneumatic brake systems. After confirming a need, ECP brake systems simultaneously send an electronic braking command to all equipped cars in the train. All cars and controlling locomotives in the train must be ECP equipped for the ECP brake system to work. ECP brakes can be installed as an overlay (OL), such that a train so equipped can be operated in ECP mode or pneumatic mode. Alternatively, ECP brakes can be installed in an ECP stand-alone configu- ration, such that the brakes on a car so equipped will respond only to ECP signals or an emergency loss of brake pipe pressure.3 According to a 2006 report by Booz Allen Hamilton, the simultaneous application of ECP brakes in response to an engineer setting the brakes on all cars in a train improves train handling during normal operations by substantially reducing stopping distances as well as by reducing longitudinal in-train forces acting along the train length as the train speeds up, slows down, or reacts to changes in grade and track curvature.4 Assessment of the potential emergency performance of the braking systems involved modeling and simulations of derailment scenarios by a private contractor (Sharma & Associates) for DOT (See Box 1). The scenarios involved a derailment event initiated while a train is on its track and during which tank cars exit the track and slide on the ground, colliding with each other and potentially with portions of the train that are still on the track. The “physics-based” results of the DOT–Sharma modeling and simulations sug- gest that trains equipped with ECP brake systems that experience an in-train separation have decreased brake application time and that the kinetic energy of tank cars leaving the tracks in a train derailment is less than that of trains equipped with pneumatic brakes alone or augmented with EOT devices or DP con- figurations.5 After consideration of those results and its own analysis, DOT concluded that the use of ECP brakes can reduce the number of cars that might derail, become punctured, and release their contents dur- ing a train accident. In comments on the proposed HHFT rule, various organizations associated with the railroad industry questioned whether the expected benefits of equipping trains with ECP brakes justify the costs of the technology.6 The organizations expressed concern that the use of ECP brakes would impose unreasonably high costs related to brake system capital investment, interoperability, reliability, and maintenance issues. They questioned whether the safety evidence was sufficient to justify the ECP brakes requirement. NATIONAL ACADEMIES COMMITTEE’S STUDY In the Fixing America’s Surface Transportation (FAST) Act,7 which was enacted in December 2015, Congress required the Secretary of Transportation to reconsider the ECP braking system requirements and determine, by the end of 2017, whether those requirements are justified. If DOT does not find that the ECP brake requirements are justified, the Secretary is expected to repeal them. In response to a congressional request in that same legislation, the National Academies of Sciences, Engineering, and Medicine (NASEM) agreed to form a committee to review the planning, execution, and results of the physical tests and related analysis that DOT will use to inform the Secretary’s decision. The committee’s statement of task is shown in Appendix A, and its membership is shown in Appendix B.8 3Additional background information on the braking systems considered by DOT is presented in the committee’s Phase 1 letter report, A Review of the Department of Transportation Plan for Analyzing and Testing Electronically Controlled Pneumatic Brakes. 4Federal Railroad Administration ECP Brake System for Freight Service. Booz Allen Hamilton, 2006. See https://www.fra.dot.gov/eLib/Details/L02964. 5Letter Report: Objective Evaluation of Risk Reduction from Tank Car Design & Operations Improvements— Extended Study. Sharma & Associates, March 2015. 6For example, see comments of AAR, Docket No. PHMSA-2012-0082 (HM-251), Sept. 30, 2014 and Jan. 30, 2016. 7Public Law No. 114-94. 8See Appendix C, Disclosure of Conflict(s) of Interest.

  3 BOX 1. Overview of DOT’s Modeling and Analysis Approacha “By working with Sharma, FRA [the Federal Railroad Administration] and PHMSA [the Pipeline and Hazardous Materials Safety Administration] have developed a model that ties together the load envi- ronment under impact conditions, with analytical/test based measures of tank car puncture resistance capacity and adjustments for suitable operating conditions, to calculate the resultant puncture proba- bilities and risk reduction. Specifically, we have provided an objective methodology for quantifying the safety performance (risk reduction) through the following approach: 1) Developing a consistent measure of the load environment associated with nominal tank car derailments through detailed analytical simulations. 2) Quantifying the puncture resistance of the given tank car designs for a nominal range of impactor sizes and impact forces, either through review of past published research (for conventional cars) or the development of new models (for innovative designs). 3) Adapting the load environment for changes in operating conditions and combining the load envi- ronment and the resistance curves with nominal impactor size distributions to evaluate the safety performance or probability of puncture for a set of designs and operating conditions. 4) Confirming the performance of the methodology through review of engineering expectations and comparisons to historical data.” aDOT Final Regulatory Impact Analysis. Hazardous Materials: Enhanced Tank Car Standards and Operational Controls for High-Hazard Flammable Trains: Final Rule. May 2015. [Docket No. PHMSA-2012-0082 (HM-251). (Page 66).] In the first phase of its task, the committee reviewed DOT’s testing and analysis plan (see Appendix F) and commented on whether it will provide objective, accurate, and reliable evaluations of the factors that DOT has identified in its comparison of the emergency braking performance of ECP brakes with that of other braking systems. The key question is whether ECP brakes would reduce the incidence and severi- ty of spills of crude oil or ethanol from derailments compared with the alternative braking systems. In this second phase of its task, the committee reviews the conduct of DOT’s tests and reports of test results and provides its findings and conclusions addressing the performance of ECP brakes relative to other braking technologies or systems tested by DOT. DOT ANALYSIS AND TEST PLAN In carrying out Phase 1 of its task, the committee considered DOT’s initial Analysis and Test Plan to Assess the Effectiveness of ECP Brakes in Reducing the Risks Associated with High-Hazard Flammable Trains (see Appendix F). The results of the analysis described in the plan were intended to update the consideration of ECP brakes in the regulatory impact analysis (RIA) for the HHFT final rule, as required by Congress in the FAST Act. The testing was intended to improve the modeling approach that DOT used in its analysis. The analysis portion of the plan addressed six aspects of calculating ECP brake effectiveness:  Assess the sensitivity of calculations of ECP brake benefits to various values of the coefficient of friction between a sliding derailed tank car and the ground until the car comes to rest.  Calculate the puncture resistance of a DOT-117 tank car from various tank material properties, such as stiffness, and validate the model by using data from full-scale impact tests.

4  Calculate the benefits of ECP brakes on a curved track that is representative of the distribution of curves on the rail network.  Analyze alternative derailment initiation events (such as broken rails or broken wheels) and model the additional scenarios if they are found to be significantly different in nature from an applied lat- eral force, which was modeled for the HHFT RIA.  Compare the sequence of events (including brake pipe pressure changes throughout the train, emergency brake applications, and locomotive engineer responses) in a derailment of a train fitted with an EOT device with that of a train outfitted with DP at the front and rear.  Validate the modeling approach used for the HHFT RIA by modeling previous real-world derail- ments and comparing the results with what happened. The plan included two types of full-scale physical testing to obtain measurement data for its analysis portion. One was a test of the puncture resistance of DOT-117 tank cars.9 The other was to determine the coefficient of friction between a sliding tank car and the ground under a variety of conditions, including wet, dry, and frozen ground (ultimately, this test was not conducted).10 COMMITTEE’S PHASE 1 REPORT The committee’s Phase 1 letter report, A Review of the Department of Transportation Plan for Ana- lyzing and Testing Electronically Controlled Pneumatic Brakes, was issued in February 2017. It discussed the need for evaluation of the performance of ECP systems relative to other braking systems through sim- ulations of derailments at multiple car positions using one or more test racks to ensure that appropriate data on the propagation of emergency ECP braking are included in comparative analyses of braking sys- tems. In addition, the report discussed the need for DOT to measure the time required for the emergency- mode application of ECP braking systems and other braking systems from initial train separation to the point at which all the brakes on the train are fully applied by testing full-scale locomotive and revenue cars that are in service (see the findings and recommendations in Box 2). The committee found that, although the factors chosen by DOT for analysis and testing are relevant to consideration of the effectiveness of ECP brakes relative to other braking systems, the department’s plan did not explain why the parameters and other factors identified in the plan were the significant ones to be considered. The report recommended that DOT ensure, before plan implementation, the focus of the plan on the significant factors worthy of testing or additional measurements. The report explained how DOT can do this by developing a multivariate regression model that makes use of two databases on train derailment accidents to examine how tank car spillage relates to specific circumstances of the accidents (see the findings and recommendations in Box 3). Since the committee considered the development of a multivariate regression model before plan implementation to be an important step, it did not provide rec- ommendations directly relevant to the execution of each task in DOT’s plan. However, in view of the im- portance of demonstrating the reliability of DOT’s modeling and simulation approach, the committee rec- ommended steps that the department should take in conducting its planned validation to demonstrate the model’s ability to simulate actual derailment events (see the finding and the recommendation in Box 4). 9DOT carried out the puncture resistance test of DOT-117 tank cars at the Transportation Technology Center in Pueblo, Colorado, on September 28, 2016, before the completion of the committee’s Phase 1 report. The objective of the test was to validate the modeling and analysis associated with the puncture resistance of DOT-117 tank cars involved in collisions following a derailment. 10On October 26, 2016, the committee chair sent a letter to Secretary Foxx indicating the committee’s concern that conducting additional tests without consideration of its advice might result in the inefficient use of limited re- sources. The committee recommended that DOT suspend its testing activities until the committee’s Phase 1 report had been issued. In response, Secretary Foxx indicated that he asked FRA to suspend preparations for tests of the coefficient of friction, which were planned for early December 2016.

  5 BOX 2. Committee’s Phase 1 Findings and Recommendations Concerning Brake Force Propagation in Emergency Applications 1.1. Finding: The DOT–Sharma modeling approach includes no delay in the emergency application of ECP brakes. All train cars are assumed to initiate emergency braking at the beginning of the simula- tion; a latency time to detect loss of brake pipe pressure and for signals to be received at all cars to initiate emergency braking was not included in the modeling. The DOT–Sharma modeling approach assumed that the ECP brake force increases linearly from zero to maximum value in 9.6 seconds. 1.2. Finding: The DOT–Sharma modeling approach interprets AAR [Association of American Rail- roads] Standard S-4200 (Paragraph 4.4.4.1, Multiple CCDs, Trailing HEUs, or EOT Has Critical Loss) as not allowing a delay for initiating the emergency application of ECP brakes. Some test rack data (see Appendix F)a exhibited a delay between the time of the loss of brake pipe pressure and the be- ginning of the rise in cylinder pressure. 1.3. Finding: The results of train car test rack simulations provided to the committee at its request rep- resented the responses of both an ECP-only [stand alone] system and an ECP-OL system. 1.4. Finding: While there was variability in the test rack results, some results suggest that during the first few seconds of emergency brake application in response to an air hose separation, initial cars approaching the point of derailment in a train with ECP brakes might experience less braking force than would cars in a train with pneumatic brakes. Some test rack results exhibited a lag during the first 1.5 seconds for ECP brake propagation followed by a rise to full pressure in the brake cylinder in 9 to 10 seconds. Although the committee does not consider the test rack results it received to be defini- tive with respect to emergency brake performance, the results point to key aspects in need of further testing. 1.1. Recommendation: To ensure that appropriate pressure–time relationships are included in com- parative analyses of braking systems, the performance of ECP systems relative to DP and EOT sys- tems should be simulated by using one or more AAR-certified test racks. The derailment-initiating event for ECP testing should comprise a synchronized separation of the air hoses and car-to-car elec- trical connectors. Tests should be conducted at multiple car positions throughout the test racks. Tests should use car control devices of both the ECP-only and the ECP-OL type. The tests should be de- signed to characterize and contrast the amount of braking power that ECP brakes and pneumatic brakes augmented with EOT devices or a DP configuration develop from the initial train separation to the point at which all of the brakes on the train are fully applied. 1.2. Recommendation: DOT should measure the time required for the emergency-mode application of ECP brakes and pneumatic brakes augmented with EOT devices or a DP configuration from initial train separation to the point at which all of the brakes on the train are fully applied by testing full-scale locomotive and revenue cars that are in service. The test should be conducted on standing trains and ones that are moving slowly. Both ECP-only brakes and ECP-OL brakes should be examined. Emer- gency braking conditions should be initiated by simultaneously disrupting the brake pipe and electric line connections. EOT and DP radio system operational delay times should be collected from field studies and then applied in test rack simulations. a Refers to Appendix F of the committee’s Phase 1 report: A Review of the Department of Transporta- tion Plan for Analyzing and Testing Electronically Controlled Pneumatic Brakes: Letter Report (2017).

6 BOX 3. Committee’s Phase 1 Findings and Recommendations Concerning Ensuring That the Testing and Analysis Are Focused on the Significant Factors 2.1. Finding: DOT’s analysis and test plan provides no justification as to why the factors chosen for consideration are the most important in predicting brake performance. 2.2. Finding: Data on train derailment accidents that have resulted in thousands of damaged tank cars are available for analysis to guide the plans for modeling and large-scale physical testing. 2.1. Recommendation: Before implementing the plan, DOT should use multivariate analysis to inform its development and to ensure that it is focused on the factors that have significant impact on model predictions. 2.2. Recommendation: FRA’s Rail Accident/Incident Reporting System (RAIRS) tank car derailment accident records with tank car–specific information (particularly concerning damage) as found in the RSI-AAR tank car accident database should be used to create a merged database for tank car derail- ment accidents. 2.3. Recommendation: A multivariate regression model should be developed from the merged data- base to examine how tank car spillage relates to a set of specific circumstances of the accidents [e.g., number of cars derailed, number of broken couplers, the nature of damage to the track structure (in- cluding the length of damaged track), the speed of the trailing locomotive at the start and during the accident (if available), and the magnitude and location of impact forces for each car in relation to the recorded postaccident damage]. The results should be examined for insights into which analysis pre- dictors are strong but are known with insufficient accuracy and therefore are worthy of testing or addi- tional measurements. BOX 4. Committee’s Phase 1 Finding and Recommendation Concerning Validation of Modeling and Simulation Approach 3.1. Finding: The ability of the modeling approach used for the HHFT final rule to simulate specific derailment events given a set of parameters was not adequately validated. Instead of examining train behavior during specific events, the validation process primarily compared simulated results for the number of cars derailed and the number of punctures against a large data scatter from historical de- railment events. 3.1. Recommendation: DOT should validate its modeling approach by comparing the model’s predic- tion of the outcome of an individual derailment event with the actual outcome of that one event. The type of braking system included in a modeled scenario should be the same as that involved in the ac- tual train derailment with which the model prediction is to be compared. Such a comparison should be conducted for a reasonably large number of real-world derailments of diverse types. Adequate phys- ics-based metrics should be the basis of comparisons between the event and the model predictions. The acquired merged database (see Recommendation 2.2) should be used for the validation tests.

  7 DOT’s ANALYSIS AND TESTING RESULTS On July 6, 2017, DOT presented the results of the analysis and testing conducted in response to the committee’s Phase 1 recommendations.11 The status of other items in DOT’s plan that were not specifi- cally addressed by the committee’s recommendations is provided in Appendix F. Brake Force Propagation in Emergency Applications In response to the committee’s recommendations on brake force propagation in emergency applica- tions (see Box 2), DOT conducted field and laboratory testing to characterize brake response profiles to be included in new simulations and analyses. Field testing was conducted on a stationary revenue train at the Norfolk Southern Corporation rail yard in Conway, Pennsylvania, to measure the latency times associated with emergency brake application initiation by a radio signal under EOT and DP systems.12 The tests involved simulating emergency brake applications at different points on a 100-car train by venting brake pipe pressure with an exhaust valve.13 The measured latency time was used in the subsequent laboratory testing, described below. In addition to measurements specified in the test plan, brake cylinder pressure buildup and the rate at which brake line pressure changed were measured during the test. Laboratory testing was conducted by using a test rack at the New York Air Brake facility in Water- town, New York, to measure brake cylinder buildup profiles and brake propagation. The test rack was set up to simulate the emergency performance of ECP brakes and conventional pneumatic air brakes in both DP and EOT configurations.14 The following actions were performed at various locations along the test rack:  Brake pipe pressure was vented to initiate an emergency brake application.  Time between brake pipe venting and loss of brake pipe pressure at each end of the train was meas- ured.  Time from hose separation until the last car reached full brake cylinder pressure was measured.  Brake cylinder pressure buildup profiles at multiple cars were collected. Revised Analysis Results The testing conducted at Conway and Watertown provided DOT with measured data on DP and EOT radio transmission latency (2 seconds), brake cylinder pressure profiles, brake propagation times, and the time difference between hose separation and brake initiation on the first adjacent car [0.17 sec- onds for conventional pneumatic systems and pneumatic systems augmented with DP or EOT systems; 0.67 seconds for ECP brakes (all cars)]. Those data were used to rerun the derailment simulations and puncture analyses. Testing results confirmed the appropriateness of a rate of propagation along the brake pipe of 950 feet per second, as had been used in the initial modeling. Table 1 presents DOT’s revised estimates of the number of derailed cars associated with the emergen- cy application of different braking systems. The presentation of results for the ECP-OL configuration, but not for the ECP stand-alone configuration, is consistent with DOT’s previous analyses, which assumed that tank cars would be equipped with overlay systems so that they could be used in non-ECP mode when appropriate. The statistics presented in the table describe the results of simulations conducted by Sharma & 11Materials in the visual aids presented by DOT to the committee on July 6, 2017, are provided in Appendix G. 12DOT did not test a train moving slowly because it did not expect to learn more concerning the objective of the plan than it would learn from testing a stationary train. 13EOT and DP Emergency Transmission Field Test Plan, Revision 3, May 22, 2017. 14ECP, EOT and DP Emergency Transmission Lab Test Plan, Revision 3, June 15, 2017.

8 Associates for a 100-car train (all DOT-117 tank cars) with a derailment initiated between the locomotives and the first car in the train. The train speed at derailment initiation was 40 mph. Eighteen simulations were conducted for each brake system in the same way as had been done previously, except for use of the new testing results concerning brake pressure propagation and the use of one speed of derailment initiation. As described by Sharma & Associates (2015), the modeling approach represents differences in derailment conditions by varying four parameters: coefficient of friction between tank cars and ground, speed of derailment initiation, severity of lateral force for derailment initiation, and quality of track.15,16 The commit- tee was not given the specific parameter values used for each model run. TABLE 1 DOT’s Estimates of Number of Derailed Cars Using Updated Brake Profiles ECP Overlay DP or EOT Conventional Average 21.0 23.1 25.5 Minimum 12 14 8 Maximum 28 39 45 Range 16 25 37 Standard deviation 4.4 5.6 7.2 NOTE: Estimated numbers of cars derailed are for 18 simulations for each brake system conducted for a train with 100 cars behind the point of derailment at a derailment initiation speed of 40 mph. SOURCE: DOT Response to NAS Letter Report on Electronically Controlled Pneumatic Brakes: Number of Cars Derailed and Punctured, dated July 12, 2017, and received from Kevin Kesler (DOT) on July 13, 2017. Table 2 presents DOT’s revised estimate of the number of punctures in derailed cars associated with emergency brake applications. DOT indicated that its modeling approach was not intended to capture puncture performance at an individual simulation level. Instead, only a single estimate of the likely num- ber of punctures is derived across all simulations for a particular braking system.17 No estimate of uncer- tainty in the likely number of punctures was provided. TABLE 2 DOT’s Estimates of Likely Number of Punctures in Derailed Cars Using Updated Brake Profiles ECP Overlay DP or EOT Conventional Likely number of punctures 4.7 5.8 6.3 Punctures per average number of cars derailed 22% 25% 25% NOTE: The likely numbers of punctures were obtained for 18 simulations for each brake system conducted for a train with 100 cars behind the point of derailment at a derailment initiation speed of 40 mph. SOURCE: DOT Response to NAS Letter Report on Electronically Controlled Pneumatic Brakes: Number of Cars Derailed and Punctured, dated July 12, 2017, and received from Kevin Kesler (DOT) on July 13, 2017. Table 3 presents the changes in the likely number of punctures between the original and revised simulations. According to the revised estimates, derailments of trains with ECP brakes are expected to result in approximately one less tank car puncture than are derailments of trains using conventional brakes augmented with DP or EOT systems. 15Letter Report: Objective Evaluation of Risk Reduction from Tank Car Design & Operations Improvements— Extended Study. Sharma & Associates, March 2015. 16DOT is reevaluating derailment scenarios by changing input parameters, including train speed and position in the train where the derailment is initiated [personal communication from Kevin Kesler (DOT), August 3, 2017]. Those results were not available to the committee during its deliberations. 17Worked Example of Puncture Probability Methodology, Kevin Kesler (DOT), personal communication, July 14, 2017.

  9 DOT provided the following reasons for changes in the likely number of punctures:  A more realistic brake propagation estimate for conventional pneumatic brakes led to a decrease in the estimated number of punctures. Previously, the maximum brake cylinder pressure buildup time allowable by AAR standards had been used in the analysis. The test rack data showed that the brake cylinder pressure buildup for the first car adjacent to the point of derailment was faster than the minimum allowed by the standard.  The estimated number of punctures also decreased for pneumatic brakes augmented with DP or EOT systems for the same reason as for conventional brakes, but not to the same extent, because of the 2-second latency in the radio signal that initiates brake application from the opposite end of the train. During the latency period, braking occurs in the conventional mode. Previously, DOT had as- sumed no latency.  The estimated number of punctures for the ECP-OL system increased after the 0.67-second delay before initiation of ECP emergency braking that was obtained from the test rack results was used. Until ECP braking is initiated (from 0 to 0.67 seconds), braking occurs in the conventional mode. Previously, DOT had assumed that there was no delay in the emergency application of ECP brakes. TABLE 3 FRA’s Original and Revised Analysis Results Analysis Likely No. of Punctures ECP Advantage over Conventional Brakes ECP Advantage over DP/EOT Conventional DP/EOT ECP-OL Original 6.6 5.9 4.3 35% 27% Revised 6.3 5.8 4.7 25% 19% NOTE: The likely number of punctures was recalculated for a train with 100 cars behind the point of derailment at a derailment initiation speed of 40 mph. DOT also provided graphs to illustrate the total brake force buildup across 100 cars of the train after initiation of a derailment for ECP brakes versus conventional pneumatic brake systems and ECP brakes versus pneumatic brakes augmented with DP or EOT systems (see Appendix H). The mean values and standard deviations in Table 1 raise a question about the effect that uncertain- ties in the model input parameters and other assumptions would have on differences between the model’s estimated numbers of derailed cars for different braking systems. The consideration of uncertainty be- comes even more important when the difference in the number of punctures between ECP-OL and DP/EOT brake configurations is as small as 1, as shown in Table 2. Ensuring That Testing and Analysis Are Focused on the Significant Factors In response to the committee’s recommendations to ensure that the testing and analysis were fo- cused on the significant factors (see Box 3), FRA consulted with members of the RSI-AAR Railroad Tank Car Safety Research and Test Project on assessing the potential for merging FRA’s RAIRS tank car derailment accident records with the RSI-AAR tank car accident database. As indicated in Appendix G, the two parties agreed that merging the FRA and RSI-AAR databases does not offer any value. However, RSI-AAR project team members identified several key variables that should be considered, including car features (such as thicknesses of tank shell) and accident details (such as train speed at the time of derail- ment) (see Appendix G). DOT indicated that all identified variables were included in DOT’s original analysis. In addition, DOT conducted a multivariate regression analysis of data from FRA’s RAIRS database. The results indicated that train speed is the most significant predictor of the severity of train derailments. Variables of secondary importance are the position in the train at which a derailment is initiated (closer to the front implies greater overall damage) and total train tonnage. Those variables were included as inputs

10 in DOT’s original approach. DOT indicated that the rest of the variation in derailment damage is attribut- able to unpredictable factors. Validation of Modeling and Simulation Approach In response to the committee’s Phase 1 recommendation on validating DOT’s modeling and simula- tion approach (see Box 4), DOT concluded that simulation of a specific derailment might not be a good predictor of model performance because accident severity varies significantly with factors other than train speed, position in the train in which a derailment is initiated, and total train tonnage. In its 2015 RIA, DOT indicates: “Derailments are highly individual and chaotic events. The intent of the model is not to replicate or predict any specific derailment, but rather to present a methodology that can be used to evaluate the relative merit of various potential strategies for mitigating damage in hazmat derailments” (page 72).18 DOT considers its modeling and simulation approach to be valid because the approach provided central estimates of the number of cars derailed or punctured from a scatter plot of data on actual derail- ment incidents that occurred at train speeds of 30, 40, and 50 mph. DOT provided additional briefing material concerning the validation of its modeling and simulation approach (see Appendix I). That material includes discussion of the following topics:  Confidence in the input parameters,  Confirmation that the trends of model prediction are in line with expectations,  Validation against physical derailment data, and  Validation against other studies. The material in Appendix I indicates that DOT compared estimates of the rear car distance traveled from a set of LS-DYNA simulations with data from the event recorder on the rear locomotive of the train de- railed at Aliceville, Alabama and found good agreement. DOT indicated that confidence in its modeling and simulation approach is based on a review of several elements of the model inputs and outputs rather than on one point of validation. COMMITTEE REVIEW OF DOT’s TEST RESULTS AND ANALYSIS Brake Testing Useful data on EOT and DP radio system latency times were obtained from the field testing con- ducted at Conway. Similarly, useful data were obtained at Watertown on the time taken to develop brake cylinder pressure of 72 pounds per square inch gauge for ECP brakes and pneumatic brakes, augmented with EOT devices or a DP configuration, from the initial train separation to the point at which all the brakes on the train are fully applied in the section of the train behind the point of derailment. As the DOT presentation indicates, the brake cylinder pressure data collected on the stationary train at Conway include the effect of the actual brake rigging on the cars and are thought to represent field conditions better than test rack data (see Appendix G). Measurements of emergency-mode brake cylinder application for ECP brakes on a full-scale locomotive and revenue cars that are in service were not con- ducted to supplement the test rack measurements. The committee did not receive final test reports documenting the measurements recorded for the various test setup conditions. In addition, the committee did not receive reports describing the subsequent data analyses used in arriving at most of the various new parameter values and their associated uncertain- 18DOT Final Regulatory Impact Analysis. Hazardous Materials: Enhanced Tank Car Standards and Operational Controls for High-Hazard Flammable Trains: Final Rule. May 2015. [Docket No. PHMSA-2012-0082 (HM-251)]

  11 ties. For example, DOT indicated that 0.67 seconds is the amount of time after derailment initiation for ECP brakes to apply. However, the measurement readings used to develop that value and the associated amount of statistical uncertainty were not provided. Identifying Significant Factors The brake pressure tests recommended by the committee in its Phase 1 report pertain only to one of several factors that might be worthy of further testing and analysis. Variables other than those concerning the various types of brake systems, for which reliable input data are available, might have important ef- fects on model results. That was the reason for the committee’s Phase 1 recommendation for multivariate regression analysis. As indicated above, DOT’s regression analysis indicated that train speed, position in the train at which a derailment is initiated, and total train tonnage are the most significant predictors of the severity of train derailments. However, on the basis of first principles, the terrain of the derailment (particularly whether the tank cars fell away from each other on a steep grade or stacked up one behind the other on level terrain) would be expected to affect the likelihood of puncture substantially. Detailed information was unavailable to the committee on how the key factors that lead to tank car derailment and puncture have been identified and included in DOT’s modeling and simulation approach, and how the stochastic nature of those and other factors contributes to the derailment and punctured car data is unclear. Model Validation Models are simplifications and approximations of the real world. They are constrained by computa- tional limitations, assumptions, and knowledge gaps. They can be expected only to help inform decisions rather than to provide comprehensive answers for making decisions. However, multifaceted evaluation activities ensure the reliability of a modeling approach for use in regulatory decision making. Among them are peer review, corroboration of results with data and other information, quality assurance and quality control checks, and uncertainty and sensitivity analyses. In carrying out its task, the committee was faced with assessing whether the behavior of DOT’s modeling and simulation approach (see Box 1) sufficiently matches the actual emergency application of brake systems. A full vetting of DOT’s modeling and simulation approach was beyond the committee’s statement of task. Thus, the committee asked DOT to conduct a validation process involving comparisons of the simulated outcome of individual derailment events with the actual outcome of the events to obtain an in- dication of validity.19 That recommendation followed from the committee’s consideration in its Phase 1 report of DOT’s preliminary effort to validate its modeling and simulation approach by qualitatively comparing the output of 18 simulations of tank car derailment accidents run at three initial train speeds with three images of actual tank car derailment pileups. The simulations’ output of the expected number of cars derailed was portrayed graphically against a background of data points from the RAIRS database (see Appendix G). The validation effort involved only derailments from the database that were within the intended domain of the simulations.20 The scatter in the model outcomes (for a given speed and train 19In its Phase 1 report, the committee did not specify which real-world derailments should be used in the recom- mended validation exercises. The committee also indicated that the appropriate number of validation exercises can- not be known in advance but that the exercises should be conducted with a reasonably large number of real-world accidents. 20In its 2015 RIA, DOT indicates: “For the purpose of validating the Dyna model, only derailments within the in- tended domain of the simulation were included, such as derailments (not collisions) of freight trains, of a minimum length, on a mainline track, etc. (specific parameter values used for this filtering can be supplied). The Dyna model was intended to simulate the type of tank car unit train derailment leading to hazmat release with the potential for significant damage, not the full range of incidents reported in the FRA database” (page 72). DOT Final Regulatory

12 length) is the result of varying model input parameters: lateral failure load of the track, ground friction value, and lateral force used to initiate the derailment. However, DOT did not demonstrate that the varia- tions in those particular parameters were appropriate, and it did not adequately substantiate the model’s key internal assumptions. The committee emphasized the need for the further validation that was indicat- ed in DOT’s analysis and testing plan (see Appendix F). After its July 2017 presentation to the commit- tee, DOT submitted a compilation of additional validation efforts (see Appendix I). The response by DOT to the committee’s request in its Phase 1 report for a validation process that involves comparisons of the simulated outcome of individual derailment events with the actual outcome of the events was a comparison with the Aliceville, Alabama, derailment, and no other. The committee finds the choice of this particular accident disappointing. The terrain of the derailment, a raised embank- ment, was not consistent with the assumptions in the model, which essentially models all derailment ter- rain as flat and featureless because of the model’s inability to consider terrain. In its Phase 1 report, the committee emphasized that the conditions of each derailment used in the validation exercises should be similar to conditions that the model can approximate, in terms of train consist and track infrastructure, including the terrain. The purpose of the recommended validation exercise was to demonstrate that the physics characterizations of the model are accurate predictors of real-world events. In the Aliceville de- railment, virtually all the cars ended up on one side of the track. Not surprisingly, the number of punc- tures that occurred exceeded the estimate of DOT’s model, which portrayed a more symmetric distribu- tion of the derailed cars (see Appendix I). DOT did not indicate whether the derailment scene dimensions are similar to or reasonably approximate the dimensions of the real accident scene. DOT also did not pro- vide a comparison between the punctured cars in the real pileup and the cars predicted by the model to have high impact loads. After consideration of the results and other information that DOT made available to the committee, several key concerns remain:  DOT’s apparent physics-based modeling approach to estimate the number of tank cars derailed does not account for the three-dimensional effects of the surrounding terrain on a derailment. The validation efforts did not use derailment conditions similar to those the model can approximate.  The model holds constant, or assigns selected values to, key stochastic factors (such as terrain geometry and soil properties) that can contribute significantly to tank car puncture. Thus, how re- liably the model relates the total brake force to the estimated numbers of tank cars derailed and punctured is unknown.  The contribution of uncertainties in the model input parameters to the uncertainties in the model’s estimated numbers of derailed cars and punctures for different braking systems was not provided.  DOT’s revised analyses provided to the committee in time for this report were only for a 100-car train with a head-end derailment at a speed of 40 mph. The committee did not have information about how the results would change if the modeled derailments were to occur farther back in the train and at different speeds.  Whether the number of simulation runs for various derailment scenarios was sufficient for estab- lishing statistically valid results was not demonstrated.  Overall, the validation process followed by DOT–Sharma does not abide by standards for verifi- cation, validation, and uncertainty quantification for computational models such as those devel- oped and widely accepted by the American Institute of Aeronautics and Astronautics, the Ameri- can Society of Mechanical Engineers, the National Aeronautics and Space Administration, and the U.S. Department of Energy.21 Impact Analysis. Hazardous Materials: Enhanced Tank Car Standards and Operational Controls for High-Hazard Flammable Trains: Final Rule. May 2015. [Docket No. PHMSA-2012-0082 (HM-251).] 21See Hills, R.G., D. C. Maniaci, and J. W. Naughton, V&V Framework, Sandia Report SAND 2015-7455, Sept 2015 (available at http://prod.sandia.gov/techlib/access-control.cgi/2015/157455.pdf) and Rider, W.J., J.R. Kamm,

  13  There was no detailed vetting process of the model simulations by outside experts who were giv- en enough data and asked to replicate independently the DOT–Sharma results. The outside ex- perts also could have examined in detail the LS-DYNA input files that DOT–Sharma used. Conclusions on the Performance of ECP Brakes Relative to Other Braking Systems Tested by DOT 1. The field testing at Conway and the laboratory testing at Watertown provided data useful in helping to ensure that appropriate brake system latency times and brake pipe and brake cylinder pressure–time rela- tionships are included in comparative analyses of braking systems. 2. The multivariate regression analysis that was conducted on FRA’s RAIRS database identified factors with a significant impact on tank car derailment and puncture estimates. However, the effect of the sto- chastic nature of those and other factors on the derailment and puncture estimates obtained from the mod- eling and simulation approach is unclear. Key parameters, such as terrain geometry, still appear to be con- sidered inadequately. 3. DOT’s efforts to validate its modeling and simulation approach in response to the committee’s request do not instill sufficient confidence in DOT’s comparison of the estimated emergency performance of ECP braking systems with that of pneumatic braking systems augmented with DP or EOT devices. DOT’s ef- forts did not follow well-established standards for the validation of computational models. 4. The committee is unable to make a conclusive statement about the emergency performance of ECP brakes relative to other braking systems on the basis of the results of testing and analysis provided by DOT. and V.G. Weirs, Verification, Validation and Uncertainty Quantification Workflow in CASL, SAND 2010-234P, Sandia National Laboratories, Dec. 2010 (available at https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/Val UQWorkflowInCASL-RiderFinal.pdf).