Letter Report on a Technical Peer Review of the Buzzards Bay Risk Assessment (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Letter Report Transportation Research Board of The National Academies Technical Peer Review of the Buzzards Bay Risk Assessment SUMMARY Over the past 72 years, eight maritime incidents in Buzzards Bay, the Cape Cod Canal, and the vicinity have resulted in significant oil spills and environmental impacts. The most recent, the Bouchard B-120 spill in April 2003, resulted in the loss of 2,333 barrels of fuel oil. These incidents have been the impetus for a number of activities at the state and federal levels to prevent and to respond to oil spills in the coastal waters of the Commonwealth of Massachusetts. Most recently, the United States Coast Guard (USCG) and the Commonwealth of Massachusetts Department of Environmental Protection (MassDEP) contracted Homeland Security Systems Engineering and Development Institute (HS SEDI), a federally funded research and development center of the MITRE Corporation, to conduct a risk assessment of Buzzards Bay and the Cape Cod Canal to identify risk mitigation measures that could be implemented. This study is referred to as the Buzzards Bay Risk Assessment (BBRA).

2 At the request of MassDEP, the Transportation Research Board (TRB) of the National Academies (TNA) appointed a committee 1 to undertake an independent technical review of the BBRA focusing on its scope, methods, and supporting data. After a review of the document and other available background information, the committee met at TNA’s J. Erik Jonsson conference facility in Woods Hole, Massachusetts, to receive briefings from MassDEP, Nuka Research and Planning Group, USCG, the HS SEDI risk assessment contract team, and stakeholder groups who attended the meeting. The committee then held follow-up closed-session deliberations. The committee believes that there are significant limitations with regard to the BBRA that bring into question its scope, methods, and data. Although it recognizes the time and budgetary constraints imposed on the BBRA, the committee believes that choices made in the formulation and execution of the study bring into question the conclusions of the risk assessment on technical grounds (e.g., use of a “change analysis” approach without clearly defining a base case, application of qualitative measures without defining the scales, reliance on experts who were neither independent nor representative of the expertise needed, and the “black box” application of economic methods). Because of these concerns, the committee believes that the ranking of the risk mitigation options in the report (i.e., additional pilots, conditional escorting based on weather, and escorting at all times) are not justified and could be reversed with slightly different and more defensible methods or assumptions. Policy decisions should not be based on this assessment. 1 See Enclosure B for the committee roster and Enclosure C for biographical sketches of committee members.

3 1. INTRODUCTION Study Objectives and Charge MassDEP requested that TRB conduct a technical peer review to evaluate the methodologies and conclusions of the BBRA. The purpose is to help ensure that the results of the BBRA provide a sound basis for evaluating the current level of federal and state regulation for Buzzards Bay and the Cape Cod Canal (see Figure 1) and for determining whether USCG should make changes in the pilot and escort system requirements codified at 33 CFR § 165.100, including the special Buzzards Bay regulations [33 CFR § 165.100(d)], when it proceeds with new rulemaking. The FIGURE 1 Map of Buzzards Bay and Cape Cod Canal. (SOURCE: MassDEP 2013.)

4 technical review of the BBRA focuses on the following key questions posed by MassDEP: 1. Is the scope of the analysis (type and extent of data gathered) sufficient to support the decisions that are being made on the basis of its results? 2. Are the methodologies that are used appropriately applied to estimate the risk reduction benefits of each alternative? (Note particularly conclusions based on the “change analysis” on page 70 and the conclusions based on the “what if” analysis on pages 71–73 of the BBRA Final Report.) 3. Do the data support the authors’ judgment and ranking of risk mitigation options? The committee held a 2-day meeting in Woods Hole, Massachusetts, in August 2013, during which it received briefings from the sponsor (MassDEP and Nuka Research and Planning Group, LLC), USCG, and the HS SEDI contract team and informal comments from stakeholder group representatives who were in attendance. The briefings allowed the committee to hear the views and concerns of state and federal representatives and other technical experts and stakeholders with regard to the underlying assumptions, methodology, data, analyses, and processes that resulted in the conclusions presented in the risk assessment. Other than the briefings, the primary inputs to the committee’s effort were the BBRA conducted by the HS SEDI contract team (HS SEDI 2013); MassDEP Comments on the Draft Buzzards Bay Risk Assessment Version 0.6, dated November 20, 2012; MassDEP’s statement to the committee (MassDEP 2013); HS SEDI’s statement to the committee; 2 the Department of Homeland Security, USCG, Regulated Navigation Area (RNA) advance notice of proposed rulemaking (USCG 2013); and information on change analysis (USCG n.d. b). After the open 2 Author’s Statement for the National Academy of Sciences, Transportation Research Board, Review of the Buzzards Bay Maritime Risk Assessment. MITRE Corporation, August 2013.

5 meeting and discussions, the committee deliberated in closed session and prepared this letter report outlining its findings and conclusions and the rationale that led to these conclusions. This letter report adheres closely to the committee’s statement of task (Enclosure D). The committee did not undertake a formal risk assessment, make specific recommendations as to what risk mitigation options should be implemented, or provide advice on techniques that must be used in future risk assessment studies. By focusing on the three items in the statement of task and the BBRA as written, the committee does not intend to imply approval or disapproval of other methods or approaches that could have been taken. This letter report is organized as follows. A brief history of the motivation for the BBRA is provided in the next section. Each of the questions outlined in the statement of task (i.e., dealing with BBRA’s scope, methods, and data) is then discussed in the following three sections. Background of BBRA USCG is responsible for developing and implementing policies and procedures that support commerce, improve safety and efficiency, and facilitate dialogue within the maritime community, with the primary goal of making waterways as safe, efficient, and commercially viable as possible. MassDEP is responsible for implementing laws and regulations concerning the prevention and remediation of oil spills in Massachusetts and its waters. Although the overall goals of USCG and MassDEP appear to be aligned, the two organizations have taken different approaches to reducing the risks associated with oil barge traffic through Buzzards Bay and the Cape Cod Canal. The different approaches led to a court case in January 2005, and litigation is ongoing. The differences and court rulings are briefly summarized in the statement of objectives for the BBRA and are reviewed below.

6 The Buzzards Bay–Cape Cod Canal navigation route is a major throughway for tank barges transporting oil to parts of the Northeastern United States. Since 1969, several tank barge groundings have resulted in the discharge of oil into Buzzards Bay, Massachusetts. Discharges of oil or other hazardous materials can adversely affect people, property, the marine environment, and the economy. In 1985 Congress designated Buzzards Bay as an estuary of national significance. Buzzards Bay also contains ecologically significant habitat for threatened and endangered species; serves as the location for sport and commercial fin and shell fishing; and provides valuable recreational uses that, among other things, promote tourism.  In 2004, the Massachusetts legislature enacted the Massachusetts Oil Spill Prevention Act (MOSPA) (as amended). Currently, MOSPA requires (a) both single- and double-hull tank barges transporting 6,000 or more barrels of oil through Buzzards Bay and the Cape Cod Canal to hire a tugboat escort to accompany them, (b) vessels towing or pushing a single-hull tank barge through Buzzards Bay and the Cape Cod Canal to have at least one licensed deck officer or tow vessel operator serving exclusively as a lookout with no other concurrent duties during the transit, and (c) the presence of additional personnel on single-hull tank barges transporting oil through Buzzards Bay and the Cape Cod Canal. (Mass. General Laws c. 21M, §§ 1, 4, 6.)  On March 29, 2006, USCG published a notice of proposed rulemaking (NPRM) to implement revisions to the RNA that was then applicable to First Coast Guard District waters. The NPRM proposed to establish additional federal navigation, safety, and waterways management improvements on Buzzards Bay.

7  On August 30, 2007, USCG published a final rule to implement the revisions that were proposed in the 2006 NPRM to amend the existing RNA for navigable waterways within the First Coast Guard District. Navigation safety measures required by these regulations can be found at 33 CFR § 165.100. The special Buzzards Bay regulations are codified in 33 CFR § 165.100(d)(5). The amended regulations established a Vessel Movement Reporting System (VMRS); required that a federally licensed pilot, who is not a member of the crew, be on board all single-hull barges transporting oil or hazardous materials through Buzzards Bay and the canal; and mandated the use of tugboat escorts for all single-hull barges transporting oil and hazardous materials through the bay and the canal. USCG prepared a categorical exclusion determination, as defined in its agency procedures for implementing the National Environmental Policy Act (NEPA), for the final rule.  In subsequent litigation, the U.S. Circuit Court of Appeals for the First Circuit found that this level of NEPA analysis and documentation was insufficient. To correct the deficiency, USCG hired a contractor to perform the necessary NEPA analysis and issued a draft environmental assessment on July 18, 2012. Despite the completion of the draft environmental assessment, differences between USCG and MassDEP as to which risk mitigation options should be implemented remained (e.g., the use of federal pilots on tank barges, the use of an escort tug system, or some type of conditional escort tug system).

8 Description of the BBRA To clarify and focus the discussion between USCG and MassDEP on particular risk mitigation options of interest in Buzzards Bay, the two agencies commissioned HS SEDI to undertake a study of the technical risks and risk mitigation options associated with oil spills in Buzzards Bay and the Cape Cod Canal. In particular, the parties were interested in evaluating the risk reduction benefits and any environmental, economic, or other quantitative or qualitative costs associated with the use of marine pilots and tugboat escorts for all vessels towing laden tank barges. 3 The study would also be used by USCG to support changes to the pilot and escort system requirements codified at 33 CFR § 165.100 [including the “special Buzzards Bay regulations” contained in 33 CFR § 165.100(d)] when it proceeded with a new rulemaking. 4 Specifically, the risk assessment report was to cover four main areas: 5  An analysis of oil spill probabilities from double-hull tank barges operating in the Buzzards Bay RNA. The analysis should consider, at a minimum, navigational safety risks, the effect of weather and environmental parameters on vessel safety, causal data from past incidents and oil spills, and published reports and data.  An analysis of the potential consequences of oil spills from double-hull tank barges operating in the Buzzards Bay RNA. The analysis should consider, at a minimum, 3 The Northeast Ocean Data Working Group maintains and updates existing data sets and adds new data sets in response to the priorities of the Northeast Regional Ocean Council, including such ocean uses as vessel traffic patterns and marine transportation. The vessel traffic patterns data include density data for different vessel types (cargo vessels, passenger vessels, tankers, and tug and tow vessels). These data and maps are available at http://www.northeastoceandata.org/. 4 The advance notice of proposed rulemaking was posted in the Federal Register on July 8, 2013. See USCG 2013. 5 See Enclosure F for a complete description of the statement of objectives for the BBRA contract between USCG and MassDEP and HS SEDI.

9 environmentally sensitive habitat and resources at risk, threatened and endangered species, seasonality associated with vulnerabilities, and limits to available oil spill containment and recovery techniques due to weather or environmental factors.  An evaluation of risk mitigation costs and benefits associated with a requirement that federally licensed pilots who are not also members of the crew be on board all double- hull tank barges transiting the RNA. The evaluation of mitigation measures should follow a recognized methodology.  An evaluation of risk mitigation costs and benefits associated with a requirement that escort vessels (tugboats) accompany all double-hull tank barges transiting the RNA. The evaluation of mitigation measures should follow a recognized methodology. The original BBRA was completed on November 30, 2012, and a revision was released on January 22, 2013. The committee acknowledges the restricted budget and schedule 6 allotted for the BBRA and the limiting contractual requirements of the tasking. The remarks that follow take these constraints into account. The committee does not try to identify minor errors or propose major new studies that would investigate barge operations from a much broader and more detailed (i.e., expensive) perspective. Instead, this report highlights assumptions, choices, and model constructs of the BBRA that, if modified, would change the BBRA’s conclusions. Some of the committee’s comments may be due to the document’s lack of transparency (e.g., the oil spill cost model is truly a “black box”). Whether by design or because of constraints, the BBRA is 6 The period of performance for the risk assessment by HS SEDI was 3 months from the date of the task order award, which was August 13, 2012.

10 relatively short, and step-by-step details of some of the analyses are not present. Thus, the committee believes that its comments are appropriate for the scope of the work. Although the organization of this report is based on the three items in the committee’s statement of task [i.e., questions of scope (Section 2), methods (Section 3), and data (Section 4)], many of the points raised below could easily be applied to multiple sections. For example, is the inclusion of distant historical data that are of limited value to the current risk assessment a problem of scope, methods, or data? It applies to all three. Therefore, the reader should not compartmentalize the items into the three sections. 2. SCOPE The committee was tasked with determining whether the scope of the analysis (type and extent of data gathered) was sufficient to support the decisions that are being made on the basis of its results. The committee believes that the scope of the BBRA was detrimentally limited in multiple ways:  The basic approach did not start with a hazards analysis.  A well-defined baseline case of existing conditions was never established.  For some parts of the assessment, the report included too much history (e.g., data to determine the frequency of spills), while for others, it was too restrictive (e.g., sufficient use was not made of the logs of available escort tugs).  There was little to no recognition of the underlying uncertainties. All of these choices led to potentially significant errors in the assessment and ranking of options. Examples of these scoping problems follow.

11 General Approach: Starting Without a Hazards Analysis First, the committee believes that a different framing of the initial approach would have been beneficial in establishing the structure of the study and justifying the conclusions. Generally, for assessments like this, a hazards (or root cause) analysis is undertaken to determine the underlying problems that are causing or creating the risks (e.g., weather, crew training, equipment malfunctions). Groundings (or collisions or allisions) are not hazards; they are consequences of a failure (hazard). A risk assessment estimates the likelihood and impact of each hazard. With this understanding of the relative importance of the underlying hazards, the assessment helps to identify potential risk mitigation options (e.g., adding sensors to measure visibility in Buzzards Bay) and to determine the effectiveness of the selected options. For which hazards would the additional pilot (or escort tug) provide a risk reduction? Groundings, collisions, and allisions could all be caused by loss of power, a distracted captain, or something else. When there is loss of power, how beneficial are additional pilots (or escort tugs)? Likewise, the selection of the event of concern (e.g., oil in the water) can limit the scope and robustness of the assessment. The decision to start the assessment with the accident (e.g., the grounding) that results in oil in the water and not the cause (e.g., the loss of power) makes the risk assessment much more difficult and potentially much less reliable. Frequency of Spill The BBRA Executive Summary (page viii) states that the “risk of oil spills was not quantitatively determined for Buzzards Bay. . . .” This statement is not entirely accurate.

12 Although only a qualitative assessment was used to determine the relative effectiveness of each mitigation measure in reducing the likelihood of a spill, a quantitative analysis was used to rank the measures on the basis of a calculated return on investment (ROI). On the basis of historical statistics, oil spills were assumed to occur with a frequency of one spill every 9 years (Section 5.1, page 41), and the cost of a spill was determined by assuming a representative heavy oil spill of 3,100 barrels or 130,200 gallons (BBRA, Section 7.1, page 85). On this basis, the ROI and “payback in years” were calculated for various mitigation measures. The committee believes that the approach taken in developing the probability of spill and the size of spill led to a higher spill probability and larger spill size than can reasonably be expected. Scarcity of accident and spill data presents challenges for any marine risk assessment, especially for tank barges that are limited in number and used principally in the United States. Common practice is to use national (or, if available, international) data to assess the probability and sizes of spills on a national or global level and then to use local data and qualitative assessment to adjust the probabilities to reflect local conditions. In this study, the probability of a spill was determined from local data only. Eight major spills have occurred in the Buzzards Bay region over the past 72 years. On this basis, the study infers a 1 in 9 probability of an annual spill. Although the report acknowledges that changes in oil transportation have resulted in improved tank barge safety, no effort was made to adjust this probability for current practice and technology. The reasons behind improvements in tank barge safety since 1990 are multifaceted and include improvements in the following: operations and management (e.g., American Waterways Operators Responsible Carrier Program), design (e.g., improved buckling strength of modern barges), regulation (e.g., double-hull requirements, which come into full effect in the United States in January 2015), technology (e.g., automatic identification system,

13 electronic charting), and waterways management (e.g., Vessel Movement Reporting System). To build the analysis on data that do not reflect the current situation is problematic. To put these improvements into perspective, trends for the number of spills and spill volume for all tank vessels (both tankers and tank barges) in worldwide trade are shown in Figures 2 and 3, respectively. The data were collected by the International Tanker Owners Pollution Federation (ITOPF). The ITOPF (2012) data for all tank vessels show a factor of 10 reduction in the number of large spills (greater than 5,000 barrels) since the mid-1970s. The total quantity of oil spillage is declining at an even faster rate, indicating that the average spill size for the larger spills is also declining. FIGURE 2 Number of spills from tank vessels worldwide, 1974–2012. (SOURCE: Data are from ITOPF 2012.) 0 20 40 60 80 100 120 1 9 7 4 1 9 7 6 1 9 7 8 1 9 8 0 1 9 8 2 1 9 8 4 1 9 8 6 1 9 8 8 1 9 9 0 1 9 9 2 1 9 9 4 1 9 9 6 1 9 9 8 2 0 0 0 2 0 0 2 2 0 0 4 2 0 0 6 2 0 0 8 2 0 1 0 N o . o f S p ill s p er Y ea r > 5,000 bbls ≥ 50 bbls and ≤ 5,000 bbls

14 FIGURE 3 Oil spill volume from tank vessels worldwide, 1974–2012. (SOURCE: Data are from ITOPF 2012.) There are similar trends for tank barges, though the data are noisier. Figure 4 presents the historical spill data given in the August 2012 report of USCG and American Waterways Operators (USCG and AWO 2012). As noted in the BBRA, there is a clear decline in the quantity of spillage, with no large spills in 2007, 2009, or 2010. 7 As described above, there are many reasons for this decline, including the transition of the fleet to double-hull vessels. In accordance with the National Research Council (NRC) study on double-hull tankers (NRC 1998), this feature alone was expected to reduce the quantity of spilled oil by approximately a factor of 4. 7 A large spill is defined as 5,000 barrels or more. 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 1 9 7 4 1 9 7 6 1 9 7 8 1 9 8 0 1 9 8 2 1 9 8 4 1 9 8 6 1 9 8 8 1 9 9 0 1 9 9 2 1 9 9 4 1 9 9 6 1 9 9 8 2 0 0 0 2 0 0 2 2 0 0 4 2 0 0 6 2 0 0 8 2 0 1 0 2 0 1 2 Q u an ti ty o f O il Sp ill ed p er Y e ar ( to n n es )

15 FIGURE 4 Oil spill volume (gallons) from tank barges in U.S. waters, 1994–2012 (42 gallons is equivalent to 1 barrel). (SOURCE: Data are from USCG and AWO 2012.) Spill volume can fluctuate significantly from year to year, since most oil spillage comes from a few large spills. Regraphing of the data presented in the BBRA report (Figure 18, page 34) shows how significant the frequency trends have been. Figure 5 shows the trends for the number of casualties 8 involving barges and the number of spills from both casualties and noncasualties. The values have been indexed, with the 2001 values set to 1.00 so that relative trends can be clearly seen. Although there has been little to no change in the annual number of casualties, the number leading to oil in the water has dropped significantly. In just 10 years, the number of spills has dropped more than 75 percent. 8 Throughout this report, for consistency with the definition used in the BBRA report, casualty refers to four types of events that can lead to an oil spill—grounding, powered grounding, collision, and allision. 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 1 9 9 4 1 9 9 5 1 9 9 6 1 9 9 7 1 9 9 8 1 9 9 9 2 0 0 0 2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 2 0 1 0 2 0 1 1 2 0 1 2 Q u an ti ty o f O il Sp ill ed p er Y e ar ( ga llo n s)

16 FIGURE 5 Relative casualty and spill frequency trends, with 2001 values indexed at 1.00. (Data are from Figure 18, page 34, of HS SEDI 2013.) Thus, one can conclude that the spill frequency estimated for the Buzzards Bay region, one spill every 9 years on the basis of a 72-year history, significantly overestimates spill probability. The figure should be reassessed, with the trends in large spill frequency for tank barges being taken into account. To estimate the cost of the representative spill, the Bouchard B- 120 spill volume was applied. This spill of approximately 2,333 barrels (98,000 gallons) (Costa 2013) did not involve a double-hull barge and is larger than the total spill volume from all tank barges operating in the United States for 4 of the past 5 years. Use of this spill likely leads to an overestimation, and the representative spill should be reassessed on the basis of actual tank barge spill-size statistics, with adjustment for the further reduction in average spill size expected for double-hull tank barges. How a more accurate assessment of the frequency and size of oil spills will affect the ranking of risk mitigation options in the BBRA is not immediately clear, but it will certainly reduce the expected damage from spills (i.e., reduce the potential “benefit” of all the options). - 0.20 0.40 0.60 0.80 1.00 1.20 2000 2002 2004 2006 2008 2010 2012 Casualties Casualties spills Non-casualty spills

17 This, in turn, will affect the cost–benefit calculations that are used to justify the options. As written, the probability and spill-size estimates are not applicable to the risk assessment. Failure to Establish an Operational Baseline for Comparison The change analysis used as the primary tool for evaluating the alternatives requires a well- defined baseline against which to measure change. However, change was not measured against such a baseline. Current operations involve only a small number of single-hull barges (less than 2 percent of the transits), and single-hull barges will be eliminated in the next 15 months. 9 Yet in some parts of the BBRA, the baseline chosen appears to assume no double-hull barges. A far more useful assumption would have been that only double-hull barges were in operation (as was requested in the original tasking given to HS SEDI). The ramifications of this choice will be discussed in the next section. Investigation of details of existing operations could have provided information for evaluating the risk mitigation options. For example, what percentage of operations occurs during adverse weather conditions? The study provided the percentage of days that had poor weather, but how did the weather affect operations, and for how many hours? The existing practice of closing the bay and canal to traffic in the event of extreme weather conditions, “closure conditions,” was not in the baseline case, nor was it addressed as a risk reduction measure. In the transition to double-hull barges, have the size and frequency of barge operations changed? Are double-hull barges larger? Do they perform better in rough seas? Understanding the answers to these types of questions is fundamental for the assessment. 9 Double-hull tank barge requirements become fully effective in January 2015, in accordance with the Oil Pollution Act of 1990. Report on the Implementation of the Oil Pollution Act of 1990. http://www.uscg.mil/npfc/docs/PDFs/Reports/osltf_report.pdf.

18 A similar problem occurs with the analysis of escort tug operations. Since January 2011, the Commonwealth of Massachusetts has imposed the requirement that an escort tug be used for all oil barges, both single and double hull, transiting Buzzards Bay and the Cape Cod Canal. There is a fairly well-documented record of escort tug activities and their interventions from January 2012 through September 2012, which creates a reasonable baseline time history for escort service in Buzzards Bay. An operations hazards analysis was not conducted on this history to determine an existing risk profile. Without establishing and understanding this baseline, changes to it are difficult to justify on the basis of the change analysis alone as described in the BBRA. This is especially problematic since some of the options being evaluated will reduce the level of safety present in current operations (e.g., changing from escort tugs for all transits to conditional tugs used only in adverse weather). Such a possibility was not considered. The operational risk reductions to human error provided by the presence of an escort tug and its crew and skipper were underrecognized. They were mentioned in the “what if” analysis and in the BBRA report Executive Summary but had no influence in the ranking exercise. While there was much discussion of the risk reductions afforded by a pilot in the wheelhouse of the primary tug providing a “second set of experienced eyes,” little credit was given to the presence of another skipper in an escort tug that meets state qualification standards and that has a different and potentially independent vantage point (being a separate vessel). It was recognized that occasionally the escort tug was called on to assist the primary tug in the handling of the barge, without direct evidence of a hazard. These events reflect the natural behavior of a primary tug skipper: “Hey, escort tug, as long as you are here, how about giving me a hand.” While this may be categorized as simple convenience, significant risk reduction is

19 realized from the behavior and the resulting operation that was not considered in the scope of the analysis. These issues are discussed in detail in the next section. Once again, a reframing of the base case that more accurately reflected current operations by including double-hull barges and escort tugs could change the rankings of the risk mitigation options recommended in the BBRA. Lack of Uncertainty or Sensitivity Analyses In any risk assessment that attempts to understand the effects of changes in operational procedures for a complex system such as oil barge operations in Buzzards Bay, uncertainty will permeate multiple aspects of the analysis. Trying to model future vessel traffic patterns, weather events, and human factors of pilots and ship captains is extremely difficult, and any analysis must acknowledge the underlying uncertainties. Estimates with regard to how effective a particular risk reduction option will be are subject to even greater levels of uncertainty. How these uncertainties will affect each of the options will vary but must be understood in evaluating the final option ranking. There is no discussion of the uncertainties involved in the assumptions, data, or models or of their impact on the final results. Inconsistent Risk Mitigation Options The decision to include the transition to double-hull barges as one of the risk mitigation options and not include it in the base case condition greatly complicated the ability to draw clear conclusions from the assessment. Likewise, the option of using an escort tug only in adverse weather was not identified for consideration in the initial task statement. Justification for adding

20 this as an option is lacking, particularly since there are no other adverse-weather-only escort tug operations in this country and none that the committee is aware of internationally. Sentinel tugs, in some applications, are deployed only in adverse weather; however, escort tugs typically escort at all times. Given that the adverse-weather-only tugs option was included in the set of options to consider, the assessment did not even begin to discuss the complexities associated with implementing such a service. The report does not define what constitutes “adverse weather,” but the criteria appear to be winds greater than 25 knots and visibility less than 0.5 mile. No thresholds were indicated for waves; they were not analyzed. Rules for determining when adverse weather conditions exist (based on some combination of weather-related variables) are not trivial. Human judgment will be relied on, and hard thresholds can be arbitrary (e.g., winds of 24.5 knots are not adverse, but winds of 25 knots are). Clearly, the ability to predict adverse weather conditions so that qualified escort tugs can be scheduled will be critical. A working group is being established to discuss options for setting weather thresholds and to investigate the associated standby costs for these tugs, but the cost–benefit calculation presented in the BBRA appears arbitrary and questionable given economic realities. Modeling a standby escort system that assumes deployment of escort tugs to a specific barge under tow when the weather deteriorates to a certain level and then predicting the risk reduction of such a system will be difficult because there is no historical basis or guideline to rely on; to the committee’s knowledge, it has not been tried before. There is a precedent for modeling the deployment of rescue tugs with weather deterioration, and some risk evaluation techniques used in those models may be applicable to this situation (Scalzo n.d.; Scalzo 1993; Scalzo and Hogue 1996). Because

21 of the novelty of this option, estimates of the costs and benefits must include associated uncertainties. In addition, some of the options evaluated are assessed relative to other options. For example, “require a sentinel or response tug in lieu of an escort tug” does not evaluate the improvement relative to the base case (single-hull fleet, no escort tugs, and no additional pilots) but to a case in which escort tugs are in operation. Likewise, “require FiFi1 firefighting capability for escort or sentinel tugs” is conditional on having the tugs in operation. There is no way to untangle these options. 3. METHODS: ANALYTIC PROCEDURES USED IN THE BBRA The committee was tasked with determining whether the methodologies used in the BBRA (“what if” and “change analysis”) were appropriately applied in estimating the risk reduction benefits of each alternative. The committee believes that though the methods could have provided the necessary analysis to evaluate the risk mitigation alternatives under consideration if they had been applied properly, decisions made in the methods’ implementation bring the BBRA’s conclusions into question:  The BBRA change analysis deviated from the prescribed USCG guidelines in significant ways.  The BBRA used an environmental damage–cost model that was designed for freshwater spill events to evaluate saltwater spills.  The change analysis as implemented relied on poorly defined and incorrectly elicited categorical factors.

22  Although many of the most critical model inputs were highly uncertain, none was subjected to sensitivity or uncertainty analyses. These decisions will be discussed in detail in this section. The BBRA refers to three main analytic tools for its assessment of risk mitigation options: change analysis (Section 6.2 of the BBRA report), what-if analysis (Section 6.3 of the report), and cost analysis (Section 7 of the report), with the change analysis being central to the study. The change analysis is used to develop a risk-scoring protocol that informs both the assessment of individual management measures’ ability to reduce risks and an overall cost– benefit analysis that compares quantified benefits of risk reduction with the costs of implementing the associated risk reduction measures. The committee finds the risk-scoring system problematic. Change analysis is a technique promoted by USCG that appears to have been developed for application in situations where distinct changes in “normal” conditions increase risks in predictable ways (i.e., ways in which the likely sources of risk associated with the changed conditions are well understood). The illustrative example in the USCG online change analysis tutorial (USCG n.d. b) concerns planning for a “tall ships” regatta that is expected to bring many spectators in small boats into the harbor where the event will be staged. The worked example in the guidelines is, by construction, narrower in scope and simpler in complexity than the Buzzards Bay problem, which may explain some of the methodological departures taken by the BBRA; real-world problems are harder. The greater complexity of the Buzzards Bay question perhaps accounts for some of the departures in the change analysis done in the BBRA compared with procedures and examples provided in the associated guidance document [e.g., the confusing use

23 of positive or negative values in “impact scores” in the BBRA (Table 14 in the BBRA report) as compared with USCG’s guidelines, where all scoring is based on more straightforward and easily understood positive-value scaling]. The change analysis procedures are detailed step-by-step in the USCG guidelines. As described there, change analysis makes extensive use of another procedure detailed in the chapter in the guidelines dealing with preliminary risk analysis (PrRA) (USCG n.d. a). PrRA is a suite of mixed quantitative–qualitative methods that lead to the calculation of risk index numbers (RINs) tied to specific hazards and measures to address them. RINs are calculated in such a way as to make them “proportional to the expected equivalent loss in dollar per year loss” (USCG n.d. a, 18), thereby facilitating evaluation of risk reduction benefits relative to the costs of implementing the measures to address the risks. This approach to cost–benefit estimation, while highly dependent on the RINs being proportional to expected equivalent dollar losses as stated above, has the advantage that “apples- to-apples” comparisons of expected benefits with implementation costs of various risk reduction measures are relatively straightforward. In addition, inherent uncertainties as to the ability of the proposed measures to reduce risks can be reflected in upper- and lower-bound estimates for the RINs and the economic benefits of implementing the measures. It is instructive to compare the final figure in the illustrative example in the PrRA tutorial (USCG n.d. a, 29) with Figure 33, “Mitigation Costs vs. Spill Cost” (page 86), at the end of the cost analysis section of the BBRA (see Figure 6). Figure 33 in the BBRA report suffers both from not showing any measure of uncertainty and from a mixing of annual costs with the costs of a single large-spill event. The BBRA figure cannot be used to assess the viability of the alternatives.

24 Subjecting prospective risk reduction measures to cost–benefit analysis (monetized value of risk reduction benefits compared with implementation and other costs associated with the risk reduction measures) is common to many risk assessment approaches, for example, the International Maritime Organization’s Formal Safety Assessment (TRB 2008). Step 5 of a typical PrRA is to evaluate the benefit of risk reduction recommendations. FIGURE 6 Comparison of two approaches for displaying the results of a change analysis: (A) example from USCG manual and (B) results of the BBRA. The BBRA can be said to constitute a hybrid change analysis in that it generally follows the recommended steps but takes a different, more qualitative and less well-grounded, approach to developing the required elements. It does not use PrRA per se. In the guidelines, accidents are characterized according to their severity levels (with outcomes in terms of specific levels of deaths and injuries and environmental and economic impacts the determinants of each level, “minor,” “moderate,” and “major”) and their frequencies of occurrence (via numerically scaled

25 likelihoods of occurrence). Incident frequencies are scored according to their (subjectively estimated) likelihood of occurrence, on a 9-point scale in the guidelines example. “Incredible” incidents are those judged by experts to have less than one chance in 100,000 of occurring in 9 years (0 on the scale); “continuous” incidents are those judged to occur 100 or more times per year (8 on the scale). While those using the PrRA approach are encouraged to “make higher level, subjective assessments of the overall frequency of each accident occurring and resulting in a specific severity level” (USCG n.d. a, 16), their valuations are aided by defined anchors and scales, in apparent contrast to the corresponding valuations in the BBRA. As noted above, the change analysis methodology depends on the analyst being able to define a clear base condition and distinct departures from that base. This is not so simple in the BBRA, as illustrated in Table 13 of that report (page 60). Here, realities appear to dictate that the base condition itself incorporate projected change that is difficult to assess in terms of its additional safety benefits, namely the mandated conversion of the entire barge fleet to double hulls by 2015. Concurrently, the potential effects of the six possible departures from current conditions presented in the table (each a proposed risk mitigation measure) are not simple to assess. For one thing, implementing multiple risk mitigation options introduces uncertain interaction effects. Although uncertainties clearly exist throughout the BBRA in the details of the analysis, uncertainty receives scant formal attention compared with the approach recommended in the guidelines. The limitations discussed above no doubt hobble many applications of the recommended change analysis approach. The defenses against these limitations implied by the tutorial (USCG n.d. b) are to define as clearly as possible points of comparison that render the change being analyzed as unambiguous as possible and to rely on rich sources of expertise to judge effects

26 with and without the risk reduction measures being examined. Such defenses appear to have been difficult to achieve in the BBRA study given the complexity of the underlying situation and the limits in time and financial resources. Cost Analysis in the BBRA: Additional Considerations As noted above, the BBRA’s cost analysis is intended to facilitate weighing the risk reduction benefits of the various mitigation options analyzed against their costs of implementation, with the estimated savings due to the avoided costs of spills being taken into account. Such a calculation requires an estimate of the changes in both frequency and severity of accidents that would result from implementation of the mitigation measures, a product of the report’s change analysis element (BBRA, Tables 19–24). The cost estimation in the BBRA relies on table lookup values developed by Etkin (2004) and presented in a conference report (referenced in the BBRA, footnotes 4 and 83). Tables in the conference report provide highly differentiated values for per gallon costs of response by various measures, further differentiated by oil type. Similarly, the tables give “socioeconomic base per gallon costs,” which are further categorized into socioeconomic and environmental per gallon costs as well as additional per gallon “environmental costs.” Values taken from these tables were used to build the cost estimates in the BBRA report (Section 7, Cost Analysis). The cost estimates were generated via “custom modification of a proprietary model” developed for the U.S. Environmental Protection Agency (EPA) for estimating spills at its regulated facilities of freshwater systems. The model was not intended for estimating the costs of individual spill incidents; too many variables need to be taken into consideration for specific spills. The Oil Spill Response Cost-Effectiveness Analytical Tool (Etkin and Welch 2005)

27 allows for more user-defined inputs on spill and location specifics and includes improvements in estimating costs when the number of events on which they are based is limited. NRC’s Marine Board study on evaluating double-hull design alternatives (TRB 2001) took a much different approach and assumed that damages would be linearly related to spill-size metrics. Thus, the Oil Spill Response Cost-Effectiveness Analytical Tool (Etkin and Welch 2005) and related cost tools that were used in the Marine Board study (TRB 2001) might be better suited for this application. In view of the above, the spill response and damage cost model used is not directly applicable to the risk assessment study in Buzzards Bay. Because of the “black box” and proprietary nature of the model, whether the values extracted from the model are upper or lower bounds cannot be determined. The model is said to quantify only “relative damage and cost for different spill types” (emphasis in original), though the paper, like the BBRA report, provides point estimates in dollars in the associated tables without further qualification. The BBRA gives a good overview of sources for cleanup and environmental cost information for oil spills. The introduction to Section 7 of the BBRA, citing a different Etkin conference report, notes that listings of per unit costs do not begin to capture the complexities involved in determining the costs associated with actual spills. But the report then appears to agree with the Etkin paper that the difficulty lies in having to resort to just “one universal per- unit cost” rather than having many differentiated values. The underlying basis for any of the unit costs listed is not elucidated in either study. The idea of attaching precise dollar estimates to “socioeconomic base per gallon costs,” as is done in the underlying reference document, is not consistent, in an obvious way, with current economic thinking on environmental injury and its effects on human well-being. Economists generally look at environmental injury in terms of its welfare impacts, relying on the

28 concept of total economic value (TEV). TEV is typically decomposed into use and nonuse values associated with environmental resources, which are further differentiated into a number of other types of values that are much discussed in the resource economics literature (e.g., option and bequest value). In standard neoclassical economics, TEV is central to the measurement of changes in human well-being associated with environmental injury or with policy interventions aimed at mitigating such harm (OECD 2006, Chapter 6). The number of gallons of oil spilled, even differentiated by oil type and sensitivity of the receiving environment as in the reference report (Etkin 2004), is not likely to be the dependent variable that economists would choose in developing estimates of the welfare changes associated with an oil spill. Gallons spilled and per gallon costs of spilled oil are used as proxies for economic damages in some instances [the Washington Oil Spill Compensation Schedule 10 (WCS) prominent among examples]. In the case of the WCS, however, the costs per gallon really are relative costs, scaled by state statute to fall between $1 and $100 per gallon. Cost modifiers from Etkin (Tables 4 and 5) are also used in the BBRA. They are more convincingly justified in Etkin’s report, where Table 5 makes the case for the ordering of the cost modifier values, albeit not the specific value differences that separate the various classes. Finally, the BBRA’s cost analysis culminates in Figure 33 (page 86), “Mitigation Costs vs. Spill Cost.” This figure is shown in Figure 6 and is misleading. Unlike the equivalent figure in the PrRA guidance document, it compares apples to oranges (see discussion above). Despite many qualifiers given elsewhere in the BBRA, the figure treats the total cost to the responsible party of the Bouchard-120 spill as a cost that occurs with a 9-year expectation. How these costs 10 “The [Washington State] Oil Spill Natural Resource Damage Assessment process (WAC 173-183) uses a compensation schedule to calculate the monetary amount of damages a spiller must pay to the state for resource restoration following an oil spill. The process allows for the reduction of the monetary damage amount based on actions of the spiller.” See https://fortress.wa.gov/ecy/publications/publications/ecy05049.pdf for more information.

29 relate to changes in welfare associated with the spill is not explained. Similarly, the true costs to society of future mitigation policies require analyses that were not done in this report. The values illustrated appear to be marginally related to the preceding analysis, and the BBRA’s approach to cost estimation, detailed above, does not generate confidence that costs and benefits have been accurately estimated. An approach that can generate upper and lower bounds for estimated costs and benefits would be preferred to what was done in the BBRA (e.g., the example from the USCG manual shown in Figure 6). Construction of the Overall Risk Scores Categorical scoring methods and ordinal scales were applied in the BBRA and are not uncommon as a means of conducting a risk assessment without a formal probabilistic analysis. Their popularity is due to their ease of use and apparent effectiveness in solving assessment problems when data are sparse or hard to interpret. However, significant concerns are associated with their use (Hubbard 2009; Hubbard and Evans 2010). In fact, the validity of these methods has rarely if ever been verified. Organizations are not set up (motivated) to track the accuracy of such analyses, especially when they are applied to risk assessments involving low-probability, high-consequence events (as is the case with oil spills in Buzzards Bay). Aside from limitations of the general approach, the way that the categorical and ordinal scales were used in the BBRA is problematic, as summarized below. The method used in the BBRA to determine the risk reduction benefit of each mitigation option (e.g., adding pilots or escort tugs) involves a multiplicative model with three ordinal- scaled factors: impact score, frequency score, and severity score.

30  The impact score has seven possible values (–5, –4, –3, 1, 3, 4, and 5), with negative scores indicating a decrease in the frequency of a risk event; 1 indicating no change in frequency; and 3, 4, and 5 indicating an increase in frequency. The reasons for this spacing of values and why “no change” was given a +1 score are unclear. Whether a –2 decreases the frequency as much as a +2 increases the frequency is not known. Descriptions of the meaning of each value were not given. The values were based on assessments from “experts.”  The frequency score uses values 2 or 3 and was determined by the historical frequency of drift groundings, powered groundings, collisions, and allisions. However, the mapping from historical frequency to frequency score is not obvious. Powered groundings, which were responsible for 13 percent of the casualties in the data set, were given frequency scores of 2, while allisions, which were responsible for 34.8 percent of the casualties, were given scores of 3. Thus, even though allisions were over 2.6 times more likely, they were given a score that was only 1.5 times larger. This unsupported mapping could easily affect the rankings of options.  The severity score uses three categories (major, moderate, and minor), with scores of 5, 3, and 1, respectively. Major severity is defined as having the economic impact of a spill exceeding $5 million, moderate is between $10,000 and $3 million, and minor is between $100 and $10,000. Values were assigned to each of the four casualty types by the historical average spill size (though the units are not given). The nonlinear rescaling of average values into three categories is troublesome, especially when no uncertainty analysis is done. The average values can be dramatically affected by one or two rare large-spill events. Since there was no attempt to indicate the range of spills associated

31 with each casualty, all collisions are scored as “major” or 5, all powered groundings and allisions are scored as “moderate” or 3, and drift groundings are scored as “minor” or 1. The three factors are then multiplied to compute a “total risk score” for each risk mitigation option for each type of casualty; negative scores indicate a risk reduction and positive scores indicate either no change in risk or an increase. The product of the three factors results in values ranging from +10 (for the “require sentinel or response tug in lieu of escort tug” option) to –50 (for the “require a pilot on all laden barges” option). Some of the resulting total risk scores are curious. For example, putting FiFi1 firefighting equipment on the tugs would greatly reduce the risk for collisions (–30) but would result in a positive value for allisions (+9). (Because “no change” in frequency is scored at +1, the value of +9 does not imply an increase in risk.) For all other options, there is little difference between the option’s effectiveness in collisions and allisions. Because of the wide intervals that define the frequency and severity scores, the ordinal ranking of their product is not assured. A “high” 2 for frequency paired with a “high” 5 for severity could in reality be worse (higher risk) than a “low” 5 for both, though the methods used would show a clear difference in the other direction. Relying on average values to set the category boundaries and having wide intervals of several orders of magnitude create identifiable problems for ranking options. To evaluate each option, the total risk scores for each casualty are summed to yield an overall risk score. These scores range from –18 for the “require sentinel/response tug in lieu of escort tugs” option to –134 for requiring an additional pilot. The scores are then color coded to indicate a preference ordering. The –134 is given a color different from that of the –120, which

32 the “switch from single to double hull” option received. Adding four multiplicative scores to achieve a single overall score is questionable (e.g., positive numbers of undetermined value are canceling negative numbers of equally undetermined value). Furthermore, in view of the uncertainties inherent in the model’s data and assumptions, whether there is a meaningful difference between overall risk scores that are separated by 10, 20, 30, or even 40 units is unclear. The approach has another problem. In the calculation of the total risk score, the various risk mitigation options only affect the impact scores; the frequency and severity scores do not change across the options. Thus, changing the barge fleet from 100 percent single hull to 100 percent double hull will not change the severity of a spill from a powered grounding (i.e., it is 3 in either case). This is wrong and misleading. As a side note, this approach does not permit an evaluation of the effectiveness of multiple risk mitigation options. The study cannot answer questions such as the following: If additional pilots and double-hull barges were mandated, what risk reduction would be achieved? This combination (or one with escort tugs too) is likely to occur, but the BBRA did not assess the risk reduction that would occur (though a stylized graph, Figure 29 on page 57, implied that it had been). Combining two options could have benefits that are greater than (or less than, or equal to) the sum of the benefits of the two options taken individually. On the basis of the methods described in the BBRA, evaluation of a combination of risk mitigation options would require a new assessment from the experts of the joint benefit of the two options. Assuming independence and additivity of risk reduction benefits is not justified.

33 Flawed Expert Judgment Elicitation Protocol Since the impact score is the only source of an “effectiveness” or risk reduction measure for each option, its value is critical to the analysis and requires solid justification. Unfortunately, this value was the only one that relied completely on expert judgment, and the expert elicitation that was done was not documented and appears not to follow any of the accepted protocols—for example, those of the Nuclear Regulatory Commission 11 or EPA 12 or those concerning oil spill risk (TRB 2008). Steps for selecting a relevant set of subject matter experts needed for the analysis, training the experts, and determining the level of agreement across the experts are not documented and may not have occurred. The actual assessment used scales (–5, –4, –3, +3, +4, and +5) that were not defined. The use of the report’s authors as the experts may be understandable in view of budgetary constraints, but it raises questions of bias. The number of escort tug operators who were interviewed and used in the assessment of the critical impact score is not presented. Interviews of the skippers of the Buzzards Bay escort tugs or interviews of skippers from one of the tanker escort systems in California, Washington, or Alaska would have added credibility to the analysis, would have provided perspective on the escort tug’s ability to reduce risk, and most likely would have influenced the recommendations. The problem that this exclusion (or limited use) of tug operations experts causes is evident in the values shown for the risk mitigation options that involve tugs. In the change analysis (Section 6.2), two escort tug options were evaluated: (a) requiring an escort tug at all times and (b) requiring an escort tug only in “adverse weather conditions or when determined 11 Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program (NUREG-1563). http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1563/. 12 Expanded Expert Judgment Assessment of the Concentration–Response Relationship Between PM2.5 Exposure and Mortality. http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_report.pdf.

34 necessary.” The impact scores (as well as the frequency and severity scores) that are used to develop the risk reduction rankings are identical for the two escort tug options. According to this assessment, there are no risk reduction benefits for escort tugs in any weather other than “adverse.” This is obviously wrong. In fact, the escort tug activity log shown in Table 12 (page 44) of the BBRA provides direct evidence to the contrary. In the table, 38 instances of escort tug support in a 9-month period are described, and only 45 percent of the instances are attributed to adverse weather (i.e., the majority of instances when escort tugs were asked to assist occurred during fair weather). In the “summary risk score” table, the two escort options were not listed separately. However, the two options do appear in the summary chart in the Executive Summary, with text discussing some undocumented stakeholder conversations. The full-time escort ranks over the adverse weather escort. Ultimately, the adverse weather escort was recommended because of presumed cost-effectiveness. Summary of Methods Discussion How correcting each of the problems noted above would affect the final ranking of options is unclear, but obviously there is no “quick fix.” Following existing guidelines for expert elicitation, devising a scoring scheme that does not conflate additive and multiplicative models, allowing risk mitigation options to affect both the frequency and the severity of a spill, and using a cost model that is not a “black box” are all necessary improvements. Unfortunately, each requires a careful reformulation of the entire approach. For example, given the current model formulation and scale values, if a risk mitigation option decreased both the frequency (with a larger negative value) and the severity of a spill (with a smaller positive value), it would appear

35 less effective (have a smaller negative total risk score) than an option that only decreased the frequency. To prevent such illogical results, careful thought must be given to the use of scales that span both negative and positive values and to a three-factor, multiplicative model. 4. DATA The committee was tasked with determining whether the data analyzed support the authors’ judgment and ranking of risk mitigation options. The committee believes that several data sets were used inappropriately, which in turn led to overconfidence in the BBRA’s conclusions. Much of the problem is attributable to the BBRA trying to take a middle road between a full- fledged probabilistic risk assessment and a purely qualitative approach. The motivation for doing so is understandable, but achieving such a middle road is difficult. The mixed approach leads to examples in which underlying data are handled in inconsistent and questionable ways. In addition, several calculations were based on misconceptions of probability models. Errors in Probability Calculations To complete the cost–benefit analysis, obviously both the costs and the benefits must be quantified (refer to Figure 6). As has been demonstrated, this necessitates the calculation of the probability of spill, the cost of a spill, and the cost of the risk mitigation options, all of which were assessed in the BBRA with suspect approaches. The committee believes that there were sufficient data to make reasonable estimates of these values provided that underlying uncertainties were accounted for. In addition, the choice of the years-between-spills metric (or its inverse, spills per year) is problematic. To account for changes in the number of transits or

36 the amount of oil shipped per year, more useful metrics would be oil spill volume per transit (or per volume of oil transported). Though it is used in critical calculations, the 9-years-between- spills estimate is not applicable to this risk assessment. A similar lack of rigor (in fact a case of misuse of risk methods) can be found in the calculation of human error probability associated with powered grounding (Section 5.2). The report uses a fault tree of possible decision points and attempts to estimate the probability of error in any given decision by counting the number of basic events and logic gates in the fault tree, treating it as the number of opportunities to make an error by the pilot (Section 5.2.1). The report then uses the reciprocal of that number as a probability of human error. This represents a major misunderstanding of the concept of fault trees, the meaning of basic events and logic gates, probabilistic concepts, and above all methods for human reliability analysis. The report states the following: The addition of an independent pilot further reduces the risk of any error in decision-making. As an independent variable, the probability of simultaneous errors by the master and pilot are multiplicative. If the decision tree represented all decisions during a transit (which it obviously does not), the probability of error from a single operator is 1/17. When the independent pilot is inserted, the probability of both operators committing the same error, is 1/17 * 1/17, or 1/289. This results in a risk probability reduction from 5.9 percent to 0.35 percent.

37 Virtually everything in this approach is wrong. First, the basis for quantification of human error probability is incorrect. By including all the “and/or” gates in the node count, failure probabilities are overcounted. Assuming that errors by the master and the pilot are independent (i.e., that the probability of error by both is the product of their individual, independent error probabilities) is not correct. The error probabilities of the master and pilot are not independent; they are likely to be correlated, since they are on the same bridge in the same environment at the same time. Distractions for one are likely to affect the other. In fact, the conditional probability of a second error given the first could approach 1.0. Thus, the implied risk reduction by a factor of about 17 (or 5.9 percent ÷ 0.35 percent) is incorrect (i.e., not based on technically sound analysis). Limited Use of Available Data Previous sections have indicated how readily available data (some of which were used in limited fashion in the report) could have improved the risk assessment (e.g., historical trends in casualties and spills). Another example pertains to weather data. Weather information plays a major role in the risk assessment. One of the better-scored risk mitigation options, conditional escort tugging, depends on the frequency of adverse weather conditions, but the analysis presented in the BBRA does a poor job of extracting relevant information from the available data sources. Critical to understanding the value of conditional escort tugs is the percentage of time that they would be used. The BBRA’s approach uses nearby New Bedford airport data and makes several assumptions about the co-occurrence of low-visibility and high-wind conditions and about the length of low-visibility periods. The report is mute on adverse wave conditions. However, several of these assumptions were unnecessary. Since nearby airport wind and

38 visibility conditions are recorded hourly and wave height and period data are available for the entrance to Buzzards Bay, the adjustments made in the BBRA can be eliminated and a detailed probabilistic model could have been constructed. See Enclosure H for a detailed weather analysis. The committee found that estimates from a better analysis of the existing data would reduce the hours needed per month for conditional escort tugs by more than half, from 72 to 31 hours. This reduction is, of course, dependent on the thresholds that are set, but with the complete hourly data set, different thresholds could be investigated. Changes of this magnitude could affect multiple calculations and could easily alter the relative ranking of conditional tugs in the report. Similarly, the escort tug logs reveal an important factor that was not sufficiently discussed in the BBRA: the major role played by economics in the use of escort tugs. From January 2011 through June 2011, MassDEP paid for normal escort tug operations, but any costs associated with assisting other vessels were charged to the barge operator. During this time, the tugs were used once (i.e., escort tugs assisted approximately 0.3 percent of transits). From July 2011 through September 2012, industry paid for all escort tug operations, including costs associated with assisting barge transits. During this 15-month period, the escort tugs were asked to assist 54 times (i.e., escort tugs assisted with 7.2 percent of transits). The difference in how the escort tugs were used under the two payment schemes (i.e., 36 times more often when fees for assistance were covered) deserves discussion. How much risk was assumed by barge operations when they handled problems on their own instead of calling in the escort tug assistance? Of the 54 assistance calls made when costs were covered, 17 involved weather- related conditions. What risk reduction was associated with these assists? There is no discussion

39 concerning what was assumed about escort tug operations when the impact scores (risk reductions) were assessed. The mitigation and spill cost analysis is hampered by the decision not to conduct a full probabilistic risk assessment. The mitigation costs for some of the options are well understood (e.g., additional pilots and escort tugs have been used for several years and accurate cost data are available); for others (e.g., using escort tugs in adverse weather), there are no data. Uncertainty of the mitigation costs should be modeled. However, more problematic is the choice of using the costs of a single historical spill in the ROI and payback period calculations. These economic analyses assume that the flawed one-spill-every-9-years estimate is certain and that the spill will be identical in damage to the 3,100-barrel Bouchard B-120 spill. Both of these estimates were based on historical single-hull data and are therefore not relevant for the double-hull barge operations in place today. Then 9 years of mitigation cost estimates are graphically compared with the single-event spill costs (see Figure 6). This approach assumes that all costs are known with certainty and that each of the mitigation options is able to prevent the spill completely. A more useful study would use spill probabilities and spill amounts that are based on double-hull barge operations and modeled as distributions to reflect the inherent uncertainties. Such an approach could dramatically reduce both values and change the economic conclusions. 5. CONCLUSIONS The committee believes that there are significant limitations with regard to the scope, methods, and data of the BBRA. The committee recognizes the time and budgetary constraints imposed on the study but believes that choices made in its formulation and execution bring into question, on technical grounds, the value and usefulness of its conclusions.

40 Statement of Task 1. Is the scope of the analysis (type and extent of data gathered) sufficient to support the decisions that are being made based on its results? The committee believes that the approach taken by the BBRA limits the value of its conclusions. Not starting with a hazards or root cause analysis puts the focus of the assessment on the consequences of failures instead of on the failures themselves. This makes the selection and evaluation of the risk mitigation options difficult. The committee believes that the failure of the BBRA to define a baseline scenario clearly (on the basis of current operations) against which risk mitigation options can be compared complicates the analysis dramatically and makes the interpretation of results nearly impossible. Risk mitigation options that were not requested in the scope of work were added (e.g., conditional escort tugs), and options that should have been included in the baseline scenario were evaluated as potential options (e.g., double-hull barges). Selection of the base data for model estimation is a necessary step in any risk assessment. The committee believes that there are several examples in the BBRA where the study relied too heavily on no-longer-relevant historical data (e.g., spill history) or did not exploit recent data that were directly applicable (e.g., escort tug logs). These decisions could easily affect the assessment’s conclusions. The committee believes that the lack of comprehensive uncertainty or sensitivity analyses is problematic. Trying to estimate the risk reduction potential of various operational adjustments in the complex oil-barge transportation system of Buzzards Bay is inherently difficult. There is no question that numerous assumptions are required to make the problem tractable. However, failure to account for the underlying uncertainties or the importance of the model assumptions

41 makes the assessment results appear unjustifiably conclusive. Decision makers relying on the assessment could place too much confidence in the risk mitigation rankings. Statement of Task 2. Are the methodologies used (“what if” and “change analysis”) appropriately applied to estimate the risk reduction benefits of each alternative? The committee believes that “change analysis” as implemented in the BBRA deviates from the prescribed USCG guidelines in significant ways. The lack of a well-defined baseline scenario and the fact that no consideration of uncertainty was modeled make the results of the change analysis approach hard to interpret correctly. The committee believes that necessary details of the change analysis protocol are missing. The steps behind the elicitation of expert judgment to assess the impact scores (the risk reduction potential) of each risk mitigation option are lacking. It appears that no tug operators were interviewed when the benefits of the escort tug options were determined. As a result, several of the impact score values appear to be wrong. The environmental damage model used in the BBRA was developed for freshwater oil spills. There is no discussion of how to adjust the model for saltwater. The committee found no discussion of how implementing multiple risk mitigation options would affect risk reduction, even though a combination of options is likely to occur. Knowledge of the interaction of the two options (e.g., what is the risk reduction of having both additional pilots and escort tugs) is necessary for determining which set of options to implement. Statement of Task 3. Does the data analyzed support the authors’ judgment and ranking of risk mitigation options considered?

42 The committee believes that the available data could have been used more appropriately. As has been mentioned, some data were pushed beyond their useful limit (e.g., spill histories), while other data were not exploited to their full extent (e.g., escort tug logs). Not using the data to determine appropriate confidence intervals results in assessments that could be given too much weight by decision makers. The committee believes that there are several examples where probability calculations were done incorrectly (e.g., fault tree analysis). Unsupported assumptions of probabilistic independence provided some risk mitigation options with too much risk reduction value. The committee believes that other data sets that were not used could have informed the study (e.g., environmental data). Estimated values (e.g., hours per year of adverse weather conditions) could have been determined with significant accuracy. The use of such data would have reduced the underlying uncertainties and improved the strength of the conclusions. In summary, the committee believes that there are significant concerns about how the BBRA was completed that bring into question the final rankings of risk mitigation options. It believes that relatively small changes, corrections, or improvements in some of the input values and assumptions could influence the ranking of options and that inclusion of the uncertainties inherent in the system could make the finality of the study’s recommendations unsupportable. Modeling uncertainties is an integral component of successful risk assessments, and failure to indicate where and how uncertainties affect the recommendations is a disservice to all users of the BBRA.

43 REFERENCES Abbreviations AWO American Waterways Operators HS SEDI Homeland Security Systems Engineering and Development Institute ITOPF International Tanker Owners Pollution Federation Limited MassDEP Massachusetts Department of Environmental Protection NRC National Research Council OECD Organisation for Economic Co-operation and Development TRB Transportation Research Board USCG U.S. Coast Guard Costa, J. 2013. Bouchard No. 120 Compared to Other Buzzards Bay Spills. Buzzards Bay National Estuary Program, East Wareham, Mass. http://buzzardsbay.org/pastspills.htm. Accessed Sept. 18, 2013. Etkin, D. S. 2004. Modeling Oil Spill Response and Damage Costs. Proc., 5th Biennial Freshwater Spills Symposium, Environmental Research Consulting, Cortlandt Manor, N.Y. Etkin, D. S., and J. Welch. 2005. Development of an Oil Spill Response Cost-Effectiveness Analytical Tool. Proc., 28th Arctic and Marine Oil spill Program Technical Seminar, Environmental Research Consulting, Cortlandt Manor, N.Y. HS SEDI. 2013. Buzzards Bay Risk Assessment, Final Report. MITRE Corporation, Va., Jan. 22.

44 Hubbard, D. 2009. The Failure of Risk Management: Why It’s Broken and How to Fix It. Wiley, Hoboken, N.J. Hubbard, D., and D. Evans. 2010. Problems with Scoring Methods and Ordinal Scales in Risk Assessment. IBM Journal of Research and Development, Vol. 54, No. 3, May, pp. 246–255. ITOPF. 2012. ITOPF Oil Tanker Spill Statistics 2012. MassDEP. 2013. Issues for Consideration and Discussion During TRB Peer Review of Buzzards Bay Risk Assessment (BBRA). Received Aug. 2. NRC. 1998. Double-Hull Tanker Legislation: An Assessment of the Oil Pollution Act of 1990. National Academy Press, Washington, D.C. OECD. 2006. Cost–Benefit Analysis and the Environment: Recent Developments. http://www.oecd.org/env/tools-evaluation/cost- benefitanalysisandtheenvironmentrecentdevelopments.htm. Scalzo, S. 1993. Experience with the Design and Operation of Escort Tugs—The Way Ahead. RINA International Conference on Escort Tugs—Defining the Technology, London. Scalzo, S. n.d. Foss Tractor Tugs in North Puget Sound Tanker Escort/Assist Service. Foss Maritime Company, Seattle, Wash. Scalzo, S., and D. Hogue. 1996. Escort Tug Performance Results. ITS ’96: The 14th International Tug and Salvage Convention, Paper No. 6. Thomas Reed Publications, Brookline, Mass. TRB. 2001. Special Report 259: Environmental Performance of Tanker Designs in Collision and Grounding: Method for Comparison. National Academies, Washington, D.C. TRB. 2008. Special Report 293: Risk of Vessel Accidents and Spills in the Aleutian Islands: Designing a Comprehensive Risk Assessment. National Academies, Washington, D.C.

45 USCG. 2013. Regulated Navigation Area; Special Buzzards Bay Vessel Regulation, Buzzards Bay, Advance Notice of Proposed Rulemaking. Federal Register, Vol. 78, No. 130, July 8, pp. 40651–40654. USCG. n.d. a. Risk-Based Decision-Making Guidelines, Volume 3, Procedures for Assessing Risks, Applying Risk Assessment Tools, Chapter 6—Preliminary Risk Analysis, 2nd ed. http://www.uscg.mil/hq/cg5/cg5211/docs/RBDM_Files/PDF/RBDM_Guidelines/Volume%2 03/Volume%203-Chapter%2006.pdf. USCG. n.d. b. Risk-Based Decision-Making Guidelines, Volume 3, Procedures for Assessing Risks, Applying Risk Assessment Tools, Chapter 7—Change Analysis, 2nd ed. http://www.uscg.mil/hq/cg5/cg5211/docs/RBDM_Files/PDF/RBDM_Guidelines/Volume%2 03/Volume%203-Chapter%2007.pdf. USCG and AWO. 2012. U.S. Coast Guard–American Waterways Operators Safety Partnership, National Quality Steering Committee, Towing Industry Safety Statistics, 1994–2011. Aug. 1. http://www.americanwaterways.com/commitment_safety/Stats_Report_1Aug12.pdf.

46