Improving Fish Stock Assessments (1998)

The major goal of the committee's simulation study was to evaluate the performance of stock assessment methods and subsets of information (fishery, survey, ageing) for simulated fish populations where the true population parameters are known and where common assumptions usually made in stock assessments are violated. This project was similar in principle to a study by the International Council for Exploration of the Sea (ICES, 1993) Working Group on Fish Stock Assessment Methods, which compared a variety of age-structured methods. However, the violations considered herein are more severe than in the ICES study.

At its meeting on January 16-18, 1996, the committee designed the simulation model (the set of parameters and assumptions) that would generate simulated data sets to be used in the study. The committee used an age-structured model to generate 30 years of commercial catch and survey information as the basis of the simulations.* Complete details describing the procedure used to generate simulated data are given in Appendix E. The 30-year data series was longer than typically available because the committee was more interested in determining assessment failures due to violations of assumptions than in studying failures caused by shortness of the time series, although the latter problem also can be experienced in actual assessments. Simulated catch-age data were produced for ages 1-15, with the age 15 group containing information for all 15+ fish. The population was affected by natural mortality and fishing mortality; fishing mortality was an increasing function of age, as described by an asymptotic selectivity function. Fishing mortality also varied over time, with fishing effort being varied to achieve desired population trends and realistic variation. Recruitment to the population was governed by an asymptotic (Beverton-Holt) spawner-recruit relationship (Chapter 3) with a large, auto-correlated, environmental error component.

Certain features were included in the simulation model to test the robustness of stock assessment methods:

1. Ageing error: Many studies have shown that ageing error is a major problem in fisheries stock assessment (e.g., Summerfelt and Hall, 1987). Mean ageing error in the simulated data sets varied from 0 at age 1 to -1 year at age 15, with increasing variation as age increased. Ageing error was included because it corrupts information contained in the age composition data about year-class progression. To simplify the analyses, ages in the age 15+

1. group were not tracked; all fish in that group had the same probability of being misaged. This is not entirely realistic because older fish in this group are probably less likely to be aged outside the 15+ group (for example, at age 14). Nevertheless, the few fish in this group relative to other ages make this a minor concern (the effects of ageing error are discussed later in this chapter).
2. Fishery catchability changes: Fishery catchability was composed of two factors; one varied as a function of time and the other as a function of abundance. The first factor increased exponentially as time passed to mimic improvements in vessel efficiency due to technological improvements and learning. The second factor was a power function of abundance with an exponent of 0.4 (a little stronger than a square-root relationship). This factor was included to simulate the hyper stability* often observed in fishery catch per unit effort (CPUE). In a stock displaying hyper stability, CPUE tends to decrease more slowly than actual population size, leading to possible stock assessment errors and risk of population collapse because there are fewer fish available for harvest than indicated by CPUE trends (Hilborn and Walters, 1992).
3. Age selectivity† differences: For three of the five simulated data sets (1, 2, and 3), increased selectivity on younger fish by the fishery occurred in the last 10 years compared with the first 20 years, as shown in Figure 5.1. This feature was included because many assessments assume constant age selectivity and because selectivity changes can mimic changes in length-age relationships. Many actual fisheries appear to have changes in selectivity. For walleye pollock in the Bering Sea, changes in selectivity have resulted from changes in fishing patterns due to learning by fishers and spatial patchiness of fish populations (Quinn and Collie, 1990). When large year classes emerge, harvesters continue to target them as they age. For Pacific halibut, there is evidence of a substantial reduction in size of fish at a given age over the past 15 years (Clark, 1996). As a result, the selectivity of young age classes has been reduced because (1) the longline gear used is less efficient at catching smaller fish and (2) a larger fraction of the young individuals are below the legal size and must be discarded (Parma and Sullivan, 1996).
4. The age of 50% maturity was much higher than the age of 50% selectivity to the fishery (Figure 5.1). The model created a population that could be quite susceptible to overexploitation because fish reproduced at a greater age than the age at which they are recruited to the fishery. Such a situation is exemplified by cod, haddock, and flounder in U.S. waters, which start to be recruited at age 1 and mature at age 2+, although the difference is more pronounced in the simulated populations.
5. The survey gear had a dome-shaped selectivity function as shown in Figure 5.1 ("survey selectivity"). This choice was made because dome-shaped selectivity and natural mortality are often confounded in stock assessment applications. In addition, one data set (3) had doubled survey catchability for the last 15 years. This feature mimicked a change in survey vessel; analysts were told that a change of vessel occurred after 15 years.
6. Most stock assessment models assume constant natural mortality. In the committee's simulations, natural mortality was constant for fish of all ages during a given year but varied from year to year; this is probably true in actual populations due to variations in predation by other species and the changing incidence of disease with age. Natural mortality was modeled as a uniform random variable between 0.18 and 0.27 (with a mean value of 0.225).
7. In some fisheries, catch statistics are inaccurate, which most likely involves underreporting (see Chapter 2). One data set (2) included underreporting of catch by 30%.
8. Various process and measurement errors in the population's dynamics and the data were included in the model for realism, including random variation in recruitment, fishery catchability, survey catchability, fishery selectivity, fishing effort, ageing error, and sampling for ages.

Five data sets were generated with the age-structured model. The simulation model was constructed in an Excel spreadsheet. The simulation procedure can be visualized as

FIGURE 5.1 Maturity, fleet selectivity, and survey selectivity for data set 1.

The five simulated data sets differed in population trend, temporal changes in fishery selectivity, underreporting of catch, and changes in survey catchability (Table 5.1). The population trend simulated either a pristine stock being fished to a low level (data sets 1-4) or a depleted stock under recovery (data set 5), two common scenarios for fisheries. Fishery selectivity either was constant over time or declined stochastically from age 5 to age 3 in the last 15 years (see Appendix E). Although this design does not include all possible combinations of the four factors, the committee believed that the analysts could not have devoted the time to additional analyses. Quantification of these factors is described in Appendix E.

By comparing results from these data sets, the effects of particular factors can be understood. Data set 4 can be considered the easiest case; it has no changes in age-specific selectivity of the fishery or survey catchability over time and no underreporting. However, it did include changes in fishery catchability over time and as a function of biomass. Data set 3 can be considered the most difficult case because it includes changes in age selectivity as well as survey and fishery catchability. A comparison of results from data sets 1 and 2 shows the effect of underreporting. A comparison of results from data sets 1 and 3 shows the effect of the change in survey catchability. A comparison of results from data sets 1 and 4 shows the effect of the decrease in fishery selectivity. A comparison of results from data set 4 with data set 5 shows the effect of a decreasing population versus a recovering population.

The true exploitable biomass and the fishery and survey indices of exploitable biomass over time are shown for each data set in Figure 5.2. Each series has been scaled by its mean to show relative patterns over time. Except for data set 3, the survey index has the same pattern as biomass; for data set 3, the doubling of catchability causes the survey index to underestimate relative biomass at the beginning and to overestimate relative biomass at the end of the period. For each data set, the fishery index does not have the same trend as exploitable biomass because of the increasing catchability of the fishery over time, the decreased age selectivity in some data sets, and the dependence of catchability on biomass. The survey index is more variable than the fishery index; the survey relative error was 30% versus 20% for the fishery.

The committee sought assistance from National Marine Fisheries Service (NMFS) analysts who regularly use the major types of stock assessment methods for real assessments. The models tested (listed in order of complexity) included a production model; two versions of a delay-difference model; and age-structured analyses using ADAPT, a spreadsheet, Stock Synthesis, and Autodifferentiation Model Builder (ADMB, a commercial

package).* The spreadsheet implementation contained features similar to the Stock Synthesis program, but was a simpler implementation of the generic age-structured assessment (ASA) model. Further details about the implementation of these methods are given later in this chapter. The committee also utilized the services of a non-NMFS expert who performed additional ADAPT analyses. Data sets were sent to analysts in mid-March 1996 (see Appendix F for transmittal letter). Although the purpose of this exercise was to compare methods, the implementation of each method was affected by complex interactions among the individuals involved in the analyses, subjective and objective modeling decisions, the base model used, and the computer implementation of the model.

In addition to the five sets of catch, age composition, CPUE, and survey data from years 1 to 30, analysts were given growth and maturity parameters (see Appendix E), the ageing error probability matrix (Richards and Schnute, 1992), and information about the structure of the population and the data. Analysts were not provided with information about natural mortality, catchability, selectivity, the recruitment process, or the amount of underreporting (although they were warned that underreporting might have occurred). Analysts were requested to perform the analyses with three combinations of abundance indices:

1. CPUE data only (coded "F")
2. Survey data only ("S")
3. Both CPUE and survey data ("B")

Analysts were asked to perform these analyses independently, that is, not to use results of one analysis to initiate others or to work with analysts using other methods.

As mentioned earlier, the survey index has a greater relative error associated with it than does the fishery index. In other aspects, the survey index is less variable because surveys either are intentionally designed to be unchanging over time (e.g., same gear, sampling design, sampling methods, and sampling areas) or are changed only with associated calibration experiments to ensure that survey data are comparable over time. Thus, catchability and selectivity for the survey data were assumed to be constant (except for catchability in data set 3). Randomized sampling designs used in surveys reduce the hyper stability effect. Conversely, commercial fishers learn and change gear, fishing areas, and fishing methods to maintain or increase their harvests. The data sets provided to analysts are representative of differences known to exist between survey and fisheries data. The committee deliberately constructed the simulation according to conventional wisdom that a survey should provide a better index of abundance than data from a fishery. Nevertheless, the committee could have just as easily interchanged the fishery and the survey to make the survey the bad source of information. Hence, the simulation should be interpreted as having contained two indices of abundance, one of which was usually a good measure of abundance, and the other not.

Analysts' results were received near the beginning of May 1996 and summarized prior to the committee's

FIGURE 5.2 True exploitable biomass, survey biomass index, and fishery CPUE for the five simulated data sets. Data are plotted as quantity divided by the mean of the quantity over the 30 simulated years.

May meeting. Analysts cautioned the committee that the time they had available for the analyses was limited compared to a normal stock assessment. They noted that other information they would normally have available for doing a stock assessment (species characteristics, reports from harvesters and biologists, and other detailed information) was not given to them in this exercise. Such factors may have compromised the ability of analysts to obtain the absolutely best estimate, and the results presented herein do not necessarily reflect real-world conditions,* but all analysts operated under the same constraints, so this exercise constitutes a reasonable comparison of

methods. One other caveat is that each data set represented the results of one replication of a stochastic process. The possibility that any given data set was extreme cannot be eliminated. Other caveats from the analysts will be discussed more completely in a planned National Oceanic and Atmospheric Administration (NOAA) Technical Memo that presents the analysts' reports.

The committee met with analysts on May 14, 1996. Each analyst presented the model results, as well as an indication of problems or insights gained in the analysis. The true values of biomass, recruitment, and exploitation were compared with analysts' estimates. The committee and analysts discussed what further work should be undertaken.

Three types of additional analyses were performed after the May meeting. First, because analysts estimated and used different values of natural mortality in the initial set of model runs (ranging from 0.15 to 0.25), the committee requested that they repeat the analyses with a common value for natural mortality equal to the true average natural mortality of 0.225. The committee asked four analysts to perform these in-depth analyses for the (1) ADAPT, (2) ASA spreadsheet, (3) Stock Synthesis, and (4) ADMB age-structured models. For each of the simulated scenarios, analysts were asked to use the three combinations of fishery and survey data, as in the previous analyses.

Second, a standard set of definitions of key management variables was agreed to and it was decided to calculate TAC (total allowable catch) in year 31 using a rate based on F_40% (as described in Chapter 4). Some analysts tried additional methods to improve the assessment; these results are reported later in this chapter.

Most existing assessment methods provide some estimate of precision of the parameter estimates. However, these are based on the structure of the assessment model being correct. Thus, unless the model structure is flexible enough to allow for major sources of uncertainty about the processes and data to be incorporated, estimates of precision tend to underestimate the true uncertainty in the assessment. It would have taken a considerable amount of additional work by both the analysts and the committee to evaluate estimates of uncertainty, so this was not done.

Finally, the committee decided to undertake retrospective analyses (see Chapter 3 for more detail about this subject) to determine the persistence of over- or underestimation of population parameters over time by the different methods. Although initial results indicated that the different methods were often able to recognize that the stock was severely depleted (or substantially recovered) at the end of the simulated time period, earlier recognition would be necessary for management to react in a timely fashion. Thus, retrospective analysis is important to determine how long it would take for the assessments to recognize underlying stock trends. The analysts were to use their methods on 16 subsets of the total data set: years 1-15, 1-16, …, 1-30.

The committee collected the results of these analyses in August 1996, summarized them, and conducted additional statistical analyses of the results to test whether trends of estimates and trends of the true values were parallel (even if biased). The committee met in August 1996 to review results obtained to that point, to formulate additional analyses of the simulation results, and to develop preliminary findings and recommendations.

This section describes the approaches taken for each specific model and presents the simulation results. The simulation results using mortality rates selected by the analysts are contrasted with results using the true M (natural mortality rate). Methods are compared on the basis of summary statistics related to important management parameters. Finally, the committee's additional analyses of model performance and retrospective analyses are presented.

The production model used is described in Prager (1994). The production model estimates productivity parameters from total harvest and indices of biomass (CPUE, survey abundance over ages). No age-structured

information was used. It was difficult for this model to produce reasonable estimates for many data set-data source combinations. In particular, no reasonable estimates could be obtained from data set 3, the most difficult one, because neither CPUE nor survey data were consistent indices of abundance over time.

Estimates by this method are shown in Table 5.2, along with the analyst's confidence in the results. Estimates were provided for MSY (maximum sustainable yield)*, F_MSY, E_MSY, B_MSY, B₃₀, and fishing mortality and biomass in year 30 relative to the MSY level (F_rel and B_rel, respectively).† The committee calculated relative fishing mortality from the fishing effort information and relative biomass from exploitable biomass values provided by the analyst.

The analyst's confidence in the results was generally low, with some moderate confidence in results from data set 4 (the easiest). When both fishery and survey data were used, estimates of MSY were 4 to 20% below the true value. However, estimates of absolute and relative fishing mortality and biomass generally were not close to the true values. Curiously, the use of fishery CPUE alone sometimes produced estimates close to the true values, even though the simulated CPUE was a biased measure of biomass. Nevertheless, the estimates often differed substantially from the true values, suggesting a lack of robustness of production models for data such as the simulated data used in this study.

The analyst cogently summarized the limitations of production models as follows: ''The simulated data sets do not seem well suited to simple production modeling, and confidence in the quantitative validity of most of the results obtained is low. Noisy data, poorly correlated CPUE and survey indices, and relatively constant effort levels all probably contribute to this situation. Without knowing the underlying population model and conducting simulations, it is impossible to say to what degree age-structure effects also contribute. The apparently high fishing mortality rates and the extensive age-structured data available suggest that these fisheries are better suited to analysis by cohort-based methods. This suggestion is strengthened by several sets of simulated data in which apparently constant fishing effort leads to a population increase and then a decrease—such a scenario is incompatible with the assumptions underlying simple production models. This suggests either environmental forcing of recruitment, nonconstant catchability, or both."

Two analysts fitted the Schnute version of Deriso's delay-difference model (Deriso, 1980; Schnute, 1985), using total harvest data, the two indices of abundance, and a recruitment index (age 5 fishery or survey data for the first delay-difference method [DD] and age 3 [data sets 1-4] or age 4 [data set 5] fishery or survey data for the second delay difference method [DDKF]). For both models, recruitment was assumed to be knife-edge (i.e., all fish are vulnerable to the fishery at the same age). The population parameters estimated from this model therefore may not be strictly comparable to the true parameters, which were calculated from the age-dependent selectivity function.

The DD method was a measurement error model. Results from this model included estimates of biomass, recruitment, and fishing mortality. Natural mortality values were the same as those used in the first Stock Synthesis method described below (0.251. 0.251, 0.169, 0.201, and 0.191 for data sets 1 to 5, respectively).

For many of the DD results (Figure 5.3), deviations in exploitable biomass from the true value showed little trend, but there are substantial deviations in absolute values over the entire time period. Not surprisingly, the use of only fishery data (left panel of Figure 5.3) produced the most extreme deviations because the fishery data had a time trend in catchability that was not incorporated in the delay-difference models. Use of fishery data tended to lead to overestimates of biomass and failure to estimate the correct trend in exploitation fraction. Better results were obtained by using survey data alone or by using both data sources, but substantial discrepancies still remained. The deviations for data set 4 were particularly surprising, because this data set was relatively well

structured. Nevertheless, the delay-difference model, by utilizing some information about age structure, showed improved estimates compared with production models. However, poor indices of abundance and the failure to use more age-structured data led to estimates that were more variable than the age-structured models discussed below. Models were rerun after the true average M value was provided, and those results are discussed in the following section.

The ADAPT approach is described in Gavaris (1988), Conser and Powers (1989), Restrepo and Powers (1991), and Conser (1993). It is essentially a cohort analysis on catch-age data where indices of abundance are included to estimate a relatively small number of parameters. The major assumption is that there are no errors in the catch-age data.

The team of analysts who met for two days to perform the ADAPT analyses found that the ageing error matrix provided by the committee was more suitable to forward types of analyses such as ASA, SS, and ADMB and decided that they could not use such information. An M value of 0.15 was used in the ADAPT analyses, based on heuristic examination of the data and an attempt to estimate M from the data after making some assumptions about selectivity and catch variability.

Generally, the procedure used was (1) to examine the data and pool it beyond some age into a plus group, (2) to obtain selectivity estimates using a separable virtual population analysis, (3) to fix the ratio of fishing mortality for the plus group to the next-youngest age, and (4) to run a nonlinear least-squares procedure to estimate population parameters. Estimates of total and exploitable biomass, recruitment, spawning stock biomass, total biomass averaged over the year (B_ave), and average exploitation fraction (Y/B_ave, where Y = yield) were calculated. Age-specific estimates of catchability were examined and trends were discovered in some data sets. Therefore, except for data set 5, fishery CPUE data were not used, despite the committee's request for these sets of analyses, because the ADAPT team did not believe the information was useful. For data set 5, this group attempted a fit of the combined survey and fishery CPUE.

The relative trend in the estimates obtained was similar to the other age-structured methods (Figure 5.4). However, estimates of abundance were negatively biased, because the choice of natural mortality of 0.15 was too low compared to the true average M of 0.225. These results confirm the general conclusion that underestimating natural mortality leads to underestimation of abundance. A consequence of underestimating abundance is a tendency to overestimate exploitation rate. The team did not compute TAC because the procedure it would have used would have taken more time than was available.

A family of age-structured assessment models is based on a statistical formulation of age-structured information and the assumption that fishing mortality is separable into age-selectivity multiplied by a full-recruitment fishing mortality (Doubleday, 1976; Fournier and Archibald, 1982; Deriso et al., 1985; Methot, 1989, 1990). A generic age-structured assessment model with these characteristics was formulated in a spreadsheet (ASA). It included a modification of the method assuming that all catch was taken halfway through the year and demonstrated that this approximation was accurate. The likelihood function consisted of a multinomial component that incorporated ageing error and a residual sum of squares term for the logarithm of each abundance index (number per boat-day from the fishery or survey catch in numbers, either pooled or by age). For data set 3, ASA assumed different catchabilities for years 1-15 and years 16-30 due to the change in survey vessel. An asymptotic curve for survey selectivity was used, but the analyst who ran ASA noted that he would have considered using a dome-shaped selectivity curve (the true situation) if he had other information to justify that choice. This spreadsheet model had similar characteristics to the Stock Synthesis model described next, but was simpler.

Estimates of natural mortality came from the Alverson-Carney procedure (based on longevity of a species and growth) and were 0.251, 0.251, 0.169, 0.201, and 0.191, for data sets 1 to 5, respectively. Trends from the survey and the fishery did not match, and the analyst suspected that the fishery data had trends in catchability. This analyst did a great deal of preliminary data exploration work to help him discover structural features in the data sets that improved his analyses. Deviations of estimated exploitable biomass from the true value were generally smaller when using only survey data or both compared to only fishery data (Figure 5.5). The ASA method usually resulted in less trend in the deviations over time than the methods previously described.

The Stock Synthesis method was implemented in a computer program as described in Methot (1989, 1990) and denoted SS-P in this report. A natural mortality value of 0.2 was used. Some models with constant recruitment were fitted to the data to mimic a production model. No results were presented from these analyses because recruitment was obviously not constant. The basic configuration of the model accounted for ageing error, used logistic curves for fleet and survey selectivity, and calculated fishing mortality values that equated the observed and predicted catch biomass values each year. The analyst calculated effective sample sizes from the mean squared error and showed that these were similar to actual sample sizes, which suggested that the SS-P

models were not overparameterized. The program used was able to project population sizes into the future under a fishing mortality policy and could readily produce retrospective analyses.

This method was based on the work of Fournier and Archibald (1982) as incorporated into a computer software product called Autodifferentiation Model Builder. Analyses using ADMB were conducted by a team composed of one NMFS scientist and a colleague from outside NMFS. ADMB is similar in character to Stock Synthesis, with some additional model features. ADMB has the capability of adding process errors in the recruitment, selectivity, catchability, and natural mortality parameters. A Bayesian framework allows results to be synthesized in terms of posterior probability distributions for selected population parameters. Because the program code is compact, it was easy to carry out retrospective analyses. Of all the models, this method was the most complex, with some models having in excess of 400 parameters. Similar to other methods, a logistic curve was used for fishery and survey selectivities in the ADMB model.

The base model (ADMB1) assumed that fishing mortality at age and year was the product of age-specific fishery selectivity, catchability, fishing effort, and a process error term. Recruitment was assumed to be random about a mean value. The objective function was the sum of a modified x² goodness-of-fit function of the age composition data, lognormal error terms for fishing effort and recruitment, and a lognormal prior distribution for M with a mean of 0.2 and a standard deviation of 0.15. In model ADMB2, additional parameters were added to the fishery selectivity function to account for changes in selectivity over time. As with previous models, results were better when survey data were used alone than when both data sets or fishery data alone were used (Figures 5.9 and 5.10). In model ADMB3, applied only to data set 1, additional parameters were added for natural mortality deviations over time.

Graphs of exploitable and total biomass for all age-structured methods conducted with estimated M values are shown in Appendix I for the five data sets. For data set 1 (Figures I.1 and I.6), most estimated series show the correct overall pattern of decline over time. The estimates of exploitable biomass are generally greater than the true biomass for the first part of the series, but most converge toward the true biomass near the end of the series. Notable exceptions are the models that use only the biased fishery data as an abundance index. It should be noted that overestimation observed in the early part of the series did not occur with total biomass (Figure I.6).

For data set 2 (Figures I.2 and I.7), the pattern of decline was captured by models that used survey data. There was a tendency for the models to underestimate exploitable biomass. This difference from data set 1 can be attributed to the underreporting of catch that was included in data set 2.

For data set 3 (the "difficult" data set, Figures I.3 and I.8), models using only fishery data or those that did not use separate parameters for the two survey vessels tended to grossly overestimate exploitable and total biomass. Other deviations could have been caused by the incorrect specification of natural mortality and changes in age selectivity over time. All models overestimated biomass at the end of the series, sometimes dramatically. No model estimated exploitable biomass particularly well for this data set, although several models captured the overall trend.

For data set 4 (the "easy" data set, Figures I.4 and I.9), most models estimated the correct trend in biomass, showing that the population was depleted at the end of the series. As in other data sets, models using only fishery data tended to overestimate biomass, particularly at the beginning of the series.

For data set 5 (Figures I.5 and I.10), the only one with an increasing trend in biomass, all models showed an increase in exploitable and total biomass over time. However, there was a general tendency to underestimate the amount of recovery in the population. Unlike the other data sets, estimates were more accurate in an absolute sense at the beginning of the series than at the end.

The delay-difference method DDKF and the four age-structured modeling methods were rerun using a true average M of 0.225. The following sections describe the results of the second set of model runs and discuss the effects of M value on stock assessment results.

The DDKF method included Kalman filtering and both measurement and process error. Results from DDKF included estimates of biomass and recruitment; true exploitation fraction was determined by dividing yield by biomass. Biomass and recruitment estimates were calculated using F and S data. For the process error model, the analyst conducted a separate investigation of the estimability of M. Results from DDKF were qualitatively similar to results from DD (Figure 5.11). One notable feature of the DDKF model was its ability to estimate accurately the amount of process error present in some data sources. This result suggests that stock assessment methods incorporating both measurement and process error reflect the uncertainty in both the data and the population more accurately.

The ADAPT method was rerun by an individual not associated with the group that performed the initial ADAPT analysis. The National Research Council's (NRC's) ADAPT method assumed (as was true) that a Gamma function describes the true shape of the gear selectivity of the simulated survey data sets and that a logistic selectivity curve describes the true shape of the gear selectivity of the simulated fishery data sets for the terminal year.

Knowing the correct true average M, the analyst who ran the ASA model attempted to estimate natural mortality from data set 1 using only the survey index. The likelihood profile was centered about 0.28, which was further from the true value than was the original value used, suggesting that the available data did not provide useful information for estimating natural mortality. One possible explanation for this result is that the use of an asymptotic survey selectivity curve, when survey selectivity was actually dome shaped, resulted in confounding natural mortality with survey selectivity parameters (Thompson, 1994). For exploitable biomass in year 1, estimates were more accurate using the true average M for the first four data sets, but not for the fifth. For exploitable biomass in year 30, estimates were closer for data sets 1, 2, and 4, but not for 3 and 5. For TAC in year 31, estimates were closer to the truth for data sets 1, 2, and 5, but not for 3 and 4. Overall, estimates were more accurate using the true average M in 10 of the 15 cases, which is not significantly different from a 50:50 ratio (x² test, p = .59).

Results from the SS-P3(S), SS-P7(F), and SS-P7(B) models with the true average M are shown in Table 5.3. For exploitable biomass, F_40%, and TAC in year 31, overall estimates were more accurate using the true average M in 28 cases, less accurate in 15 cases, and the same in 2 cases; thus, the true M yielded significantly more accurate values for these three parameters (x² test, p { .05).

A final ADMB model (ADMB4) was constructed, using the true average M and incorporating time-series variations in fishery selectivity, fishery catchability, and natural mortality. For data set 3, the analyst chose not to incorporate two selectivity curves for the survey but rather allowed for time-specific deviations in catchability in the model and examined whether a trend in survey catchability occurred. Results are shown in Figure 5.12.

The statistics provided by stock assessments that are most useful for management include recent estimates of biomass, the amount of historical decline, recommended catch limits (TACs), recommended exploitation fractions, and spawner and recruitment information such as average spawning biomass and average recruitment. Specifically, the following information was summarized:

• Exploitable biomass in year 30 (EB₃₀)
• Ratio of BE in year 30 to EB in year 1 (EB₃₀/EB₁), a measure of population decline or building
• Total allowable catch in year 31 from an F_40% exploitation strategy (TAC₃₁)
• Ratio of TAC₃₁ to projected exploitable biomass in year 31 (TAC₃₁/ ₃₁), the projected exploitation rate. Projected exploitable biomass in year 31 was obtained by multiplying each analyst's estimate of exploitable biomass in year 30 by the true ratio of EBs in years 31 and 30. This was done to determine the recommended exploitation fraction (TAC₃₁/₃₁) in a consistent manner for all models, because some analysts did not provide estimates of EB in year 31.
• Average recruitment ( ) from years 1 to 30
• Average spawning biomass ( )

Results for these management parameters are expressed in terms of relative error [(estimated - true)/true] for the various models, and results are shown separately for runs made with estimated versus the true M value in tables that follow. For the purpose of evaluating different models a modest goal for stock assessment is to obtain estimates of key management parameters within ±25% of the true values.

A summary of the results across data sets is presented to show the general performance of the models. The summary statistic used was the average of the absolute values of relative errors among the five data sets. The average of absolute relative errors is used to illustrate the precision of estimates compared to true values and the average of relative errors are used to illustrate the accuracy or bias. The absolute value indicates models that deviated in either direction from the truth (Table 5.4); results from individual data sets are presented in Tables 5.5 to 5.9 and can be used to examine the direction of the errors for the individual sets. Models that did not have results for all five data sets were excluded.

Results given in Table 5.4 show that there can be great deviations in management statistics from the true values. Figure 5.13 shows the results for exploitable biomass in year 30, with relative absolute deviations often exceeding 50%, comparing estimated M versus the true average M. Model runs that used only fishery data generally yielded the worst results, those that used only survey data were generally the best, and those that used both sources ranged from best to intermediate in their results. The success of runs with both data sources depended greatly on which model was used. Model runs with more complex treatment of fishery data (SS-P6, SS-P7, ADMB4) tended to have lower errors than those with simpler treatment (DD, ASA, SS-P3, NRC ADAPT), and a similar result was obtained when fishery data were used alone. Results obtained with the true average M were not much different from those with estimated M values, probably because the true average M used in the revised analyses was, in many cases, not that much different from the M used initially. The fact that the true population had varying M over time whereas analysts used constant M (except ADMB4) may have been more important than changes in average M.

Results for the ratio of biomass in year 30 to that in year 1 were similar to those for EB₃₀ (compare Table 5.4B). This result was somewhat surprising because the committee expected it would be easier to estimate the overall change in populations than their absolute biomes levels.

The committee thought it was very important to consider the relative error in TAC, because this statistic is frequently the end point of assessment, being the recommended catch level to be taken in the next year.* The

FIGURE 5.13 Average absolute values of relative errors for exploitable biomass in year 30. (Results prior to [upper panel] and after [lower panel] knowing the true M.)

relative error in TAC was comparable to the relative error in exploitable biomass, sometimes higher and sometimes lower (see Table 5.4). In many cases, it deviated more than 50% from the true value. As before, models using only survey data or both data sources fared better than those using only fishery data. The ratio of TAC to exploitable biomass (TAC₃₁/₃₁) is the recommended exploitation rate. The amount of relative error in this statistic was lower overall than for the previous statistics, especially for model runs using only fishery data (Table 5.4). Many model runs yielded relative errors well below 50%. For this statistic, the model used seemed also to be a factor: the NCR ADAPT and ADMB4 models had greater errors than other models, regardless of which combination of data was used. Average recruitment () and average spawning biomass () were estimated with less error overall than other management parameters. This is not surprising because these statistics are calculated over the entire time period, which averages out the positive and negative errors found in individual years.

Tables 5.5 to 5.9 present the relative errors in management variables for data sets 1 to 5, respectively. These tables show that average recruitment () and average spawning biomass () are estimated with less error overall than other management parameters. The ADAPT model generally underestimated these because the natural mortality used was too low. Models using only fishery data (F) often overestimated average biomass, unless the models contained sufficient additional parameters to overcome the catchability patterns. For data set 2, many models underestimated and because of underreporting. The same thing occurred with data set 5 because biomass and recruitment were underestimated for the recovering population. For the first four management parameters related to exploitable biomass and TAC, the amount of error is quite variable among parameters and among models. The worst results are generally obtained by using only fishery data for all data sets. However, the tendency for most models using survey data to underestimate biomass in data set 5 results in the fishery-only models sometimes producing estimates that are closer to the truth for this data set. The underestimation could have been due to the particular set of random numbers used for this data set, so it would not be appropriate to conclude that assessments based on commercial CPUE generally would be better.

For data set 1, examination of Table 5.5A shows that the goal of being within 25% of the true value was achieved for at least two of the first four management parameters (related to exploitable biomass and TAC for the DD(S), SS-P3(B), SS-P3(S), SS-P6(B), and SS-P7(B) models [notation indicates model followed by data set in parentheses]). In only one case (SS-P7[B]) however, was this goal obtained for all four management parameters. For data set 2, the 25% criterion was achieved for at least two parameters with the DD(S), SS-P3(S), SS-P6(B), SS-P6(F), and SS-P7(B) models, but no model met the criterion for all four parameters (Table 5.6A). For data set 3, the ''hard" data set, the criterion was achieved for two parameters only for the DD(S) and SS-P7A(B) models, and neither of these models achieved the criterion for all four parameters (Table 5.7A). For data set 4, the so-called easy data set, the criterion was achieved for the DD(S), AD(F), AD(B), SS-P3(S), SS-P3(F), SS-P3(B), SS-P6(F), SS-P6(B), SS-P7(F), and SS-P7(B) models, but only SS-P3(S) achieved the criterion for all four parameters (Table 5.8A). For data set 5, the criterion was achieved for at least two parameters by the DD(F), DD(S), ASA(S), SS-P6(F), SS-P6(B), SS-P7(F), and SS-P7(B) models (Table 5.9A). The criterion was met for all four management parameters only for the ASA(S) and the SS-P7(F) models. The DD(S) model met the criterion for at least two parameters for all five data sets. Otherwise, no single model performed superlatively across all data sets and all management parameters.

Results using the true average M value are presented in Tables 5.5B to 5.9B. By using the true average M value, and are estimated with even less error overall than results obtained without knowledge of the true average M. This probably results from a combination of use of the true average M and use of more complicated models in the second round of model runs that corrected some biases in the earlier models. However, in some cases, using the correct M in a model did not result in an improvement. For example, for data set 3, the second-run models tended to overestimate average recruitment, which did not occur with the first run. Whether this was due to the variability in M, confounding of model parameters, or random chance is unknown.

For data set 1, relative errors were less than 25% for at least two of the exploitable biomass and TAC statistics for the NRC ADAPT(S), SS-P3(S), SS-P3(B), SS-P6(B), SS-P6(F), and SS-P7(B) models; only the latter model met this criterion for all four parameters (Table 5.5B). For data set 2, at least two statistics achieved the criterion for the ADMB4(B), NRC ADAPT(S), SS-P3(S), SS-P3(B), SS-P6(F), SS-P6(B), and SS-P7(B) models. No model achieved the criterion for all four parameters, because the underreporting led to underestimation of biomass with resultant effects on TAC (Table 5.6B). For data set 3, the 25% criterion was achieved for two statistics only by the DDKF(S) model (Table 5.7B). As with the prior results, there was a general tendency to overestimate biomass and TAC statistics for this data set, often by a wide margin. For data set 4, all models (except DDKF and

NRC ADAPT) had at least two statistics within 25% of the truth. All four statistics met the criterion for the ADMB4(B), SS-P3(S), SS-P3(B), SS-P6(B), and the SS-P7(B) models (Table 5.8B). For data set 5, all models achieved the criterion for at least two statistics, except for the ADMB4(B), NRC ADAPT, SS-P3(B), and SS-P3(F) models (Table 5.9B); SS-P6(F), SS-P6(B), SS-P7(B), and SS-P7(F) fell within 25% for all four statistics.

Overall, Tables 5.5B-5.9B show that errors in exploitable biomass and TAC are less than 25% (bold entries) more frequently for complex models (defined here as ADMB4, SS-P6, and SS-P7) than for simple models (defined here as DDKF, NRC ADAPT, ASA, and SS-P3); bold entries for estimates of exploitable biomass occur for 23% (44%) of simple (complex) model runs; bold entries for estimates of TAC occur for 17% (35%) of simple (complex) model runs. Table 5.10 suggests some modest improvement in management parameters by having the true average M; for example, the percent success (as defined in Table 5.10) was greater for two to four of the five data sets (depending on the variable) when M was set to its true mean. Yet, it should be noted that no one model performed superbly in all cases and for all management parameters.

The results from these comparisons suggest that having the correct value of M did not significantly improve the assessment results for the ASA and SS models used before and after the true average M value was revealed. This conclusion appears to be independent of which summary statistic or which data sources are used in the assessment. The reason for this modest effect of incorrect M on all models is probably that variability in M, ageing

error, changes in catchability and selectivity, and random variability dominated assessment results more so than the choice of a specific M value. The other reason is that many analyses used an estimated M value not too far from the true average M. One notable exception was the application of the ADAPT model to these simulated data sets. Dramatic underestimation of biomass occurred using an M of 0.15. When the correct average M was used, the NRC ADAPT results were better than the original ADAPT results but still showed major departures from the true values. Therefore, other factors still dominated the assessment results.

The committee expected that relative measures of biomass and exploitation would be estimated more accurately than absolute quantities. Examination of Tables 5.5 to 5.9 shows that this expectation was not fulfilled often. The estimated decline or increase in population over the time period (EB₃₀/EB₁) often had as much or more error than the most recent estimate of exploitable biomass (EB₃₀). Similarly, the projected exploitation fraction in year 31 (TAC₃₁/₃₁) frequently had greater error than the projected catch limit (TAC₃₁) itself. When this occurred, the reason was often that exploitable biomass was underestimated but TAC was overestimated, which resulted in even greater error in the exploitation fraction.

Visual comparison of biomass trends from the assessment models in Appendix I (Figures I.1 to I.10) shows that in many cases the lines (1) do not coincide with the true values and (2) are not parallel with true values. For example, the trends in exploitable and total biomass estimated by ADMB1(F) for data set 2 (Figures I.2 and I.7)