Improving Fish Stock Assessments (1998) / Chapter Skim
Currently Skimming:
5 Simulations
5 Simulations
Pages 59-110
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 59... ...
Certain features were included in the simulation model to test the robustness of stock assessment methods: 1. Ageing error: Many studies have shown that ageing error is a major problem in fisheries stock assessment (e.g., Summerfelt and Hall, 1987~.
|
From page 60... ...
6. Most stock assessment models assume constant natural mortality.
|
From page 61... ...
The committee sought assistance from National Marine Fisheries Service (NMFS) analysts who regularly use the major types of stock assessment methods for real assessments.
|
From page 62... ...
Analysts were asked to perform these analyses independently, that is, not to use results of one analysis to initiate others or to work with analysts using other methods. As mentioned earlier, the survey index has a greater relative error associated with it than does the fishery index.
|
From page 63... ...
May meeting. Analysts cautioned the committee that the time they had available for the analyses was limited compared to a normal stock assessment.
|
From page 64... ...
First, because analysts estimated and used different values of natural mortality in the initial set of model runs (ranging from 0.15 to 0.25) , the committee requested that they repeat the analyses with a common value for natural mortality equal to the true average natural mortality of 0.225.
|
From page 65... ...
Nevertheless, the estimates often differed substantially from the true values, suggesting a lack of robustness of production models for data such as the simulated data used in this study. The analyst cogently summarized the limitations of production models as follows: "The simulated data sets do not seem well suited to simple production modeling, and confidence in the quantitative validity of most of the results obtained is low.
|
From page 66... ...
They are included in this table for scientific interest. The major characteristics of the five data sets are given in Table 5.1 True values are those calculated by the committee from known parameter values.
|
From page 67... ...
Deviations greater than 400% are not shown in this figure and Figures 5.4 - 5.12. The major characteristics of the five data sets are given in Table 5.1.
|
From page 68... ...
However, estimates of abundance were negatively biased, because the choice of natural mortality of 0.15 was too low compared to the true average M of 0.225. These results confirm the general conclusion that underestimating natural mortality leads to underestimation of abundance.
|
From page 69... ...
SIMULATIONS 69 Survey 0 0 ._ -1 00 -200 -300 -400 0 ._ 5 -100 -200 -300 -400 0 0 ._ -1 00 -200 -300 -400 0 0 ._ -1 00 -200 -300 -400 0 ._ 5 -100 -200 -300 -400 0 5 10 15 20 25 30 Year FIGURE 5.4 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for the ADAPT model.
|
From page 70... ...
70 400 o ._ t'5 300 ·5 ~ 200 41 00 o 400 o ._ t'5 300 ·5 200 1 00 o 400 o t'5 300 ·5 ~ 200 4100 o 400 o ._ t'5 300 .o ~ 200 4100 o 400 300 200 4100 o IMPROVING FISH STOCK ASSESSMENTS Fishery Survey Both 0 5 10 15 Year 25 Year Year FIGURE 5.5 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model ASA.
|
From page 71... ...
, models using only fishery data or those that did not use separate parameters for the two survey vessels tended to grossly overestimate exploitable and total biomass. Other deviations could have been caused by the incorrect specification of natural mortality and changes in age selectivity over time.
|
From page 72... ...
_-,- __ 811_ ~C O 400 lo ._ ^^ ~T~ our ~1 200 _ ~ ~ _ 1 00 ~A ~1 O ~:~:~:~ 400 O n5 300 ·5 200 .~1 ~1~'~ O 100 .~ n ~ ~ ~_~,~ v ~-I-~_ ~ HI 400 o ._ ~Our .o 200 ~ 1001~ j C ~_~/~/ ~_~ Ho_ _ i __ _1 ~ ~ ~ ~ _ _ ~ . ~ , ~- _ U ~ ~ I -- ~ 115;~ - _~ Ail -~ -em _~ lilting To_ ~ ~ ~ 400 o ._ ~ ~ OUU ~ .o En 200 an I u u O ~ _ ~ ~ ^_ ~ ~_ _ ~e ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~-~ ~ ~ ~ ~ ~ ~ ~::::: ::~::::::::~:~:~:~:~::: : ~::: :~ 0 5 10 15 20 25 30 0 5 10 15 20 25 Year Year 30 0 5 10 15 20 25 30 Year FIGURE 5.6 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model SS-P3.
|
From page 73... ...
SIMULATIONS Fishery 400 o I' 300 ·5 200 100 o 400 300 200 100 o 400 300 200 100 o 400 o I' 300 ·5 200 100 o 400 o I' 300 ·5 200 100 o Survey 20 Year 30 0 5 10 In It Cal In It 15 20 25 30 Year FIGURE 5.7 Percent relative deviations of estimated biomass from true exploitable biomass for model SS-P6.
|
From page 74... ...
74 400 300 200 o 1 00 o 400 300 200 o 1 00 o 400 300 200 o 1 00 o 400 300 200 o 1 00 o 400 300 200 o 1 00 o IMPROVING FISH STOCK ASSESSMENTS Fishery Both 5 10 15 20 25 Year 30 0 5 10 15 20 25 30 Year FIGURE 5.8 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model SS-P7.
|
From page 75... ...
SIMULATIONS 400 300 200 41 00 o 400 o ._ t'5 300 ._ 200 1 00 o 400 300 200 4100 o 400 o ._ t'5 300 . _ ~ 200 4100 o 400 300 200 4100 o 75 Fishery Survey Both 0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30 Year Year Year FIGURE 5.9 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model ADMB 1.
|
From page 76... ...
76 IMPROVING FISH STOCK ASSESSMENTS Both 400 300 200 100 o 400 300 200 100 o 400 300 200 100 o 400 300 200 100 o 400 300 200 100 o 0 5 10 15 20 25 30 Year FIGURE 5.10 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model ADMB2.
|
From page 77... ...
that a Gamma function describes the true shape of the gear selectivity of the simulated survey data sets and that a logistic selectivity curve describes the true shape of the gear selectivity of the simulated fishery data sets for the terminal year. Separable ASA Models Knowing the correct true average M, the analyst who ran the ASA model attempted to estimate natural mortality from data set 1 using only the survey index.
|
From page 78... ...
78 400 o ._ I' 300 ·5 200 o 1 00 o 400 o ._ tt' 300 ·5 200 o 1 00 o 400 o ._ tt' 300 ·5 200 o 1 00 o 400 o ._ I' 300 ·5 200 o 1 00 o 400 o ._ I' 300 ·5 200 o 1 00 o IMPROVING FISH STOCK ASSESSMENTS Fishery Survey 0 5 10 15 20 25 30 0 5 10 Year 15 20 25 30 Year FIGURE 5.11 Percent relative deviations of estimated exploitable biomass from true exploitable biomass for model DDKF.
|
From page 79... ...
0.2 3571 0.225 4258 5158 0.2 0.159 0.151 0.225 0.173 0.172 0.182 0.182 0.2 447 0.225 527 737 37885390 45765590 51585158 0.160 0.163 0.182 627 681 737 439 565 737 NOTE: Results are given for three models: Model SS-P3 with only survey data, model SS-P7 with only fishery data, and model SS-P7 with both data sources. Estimates compared are exploitable biomass in year 30 (EB30)
|
From page 80... ...
80 400 300 200 41 00 o 400 o ._ t'5 300 ._ ~ 200 41 00 o 400 300 200 4100 o 400 o ._ t'5 300 . _ ~ 200 4100 o 400 300 200 4100 o IMPROVING FISH STOCK ASSESSMENTS Fishery Survey Both 0 5 10 15 20 25 30 0 5 10 15 20 25 Year Year FIGURE 5.12 Percent relative deviations of predicted and true exploited biomass for Model ADMB4.
|
From page 81... ...
for the various models, and results are shown separately for runs made with estimated versus the true M value in tables that follow. For the purpose of evaluating different models a modest goal for stock assessment is to obtain estimates of key management parameters within +25% of the true values.
|
From page 82... ...
/truel] for Important Management Parametersa Results with M= 0.225 Model Data source EB30 EB30 TAC31 TAC3 ~R S DDKF F 5.62 3.89 8.49 0.42 0.36 na DDKF S 0.51 0.32 0.81 0.30 0.22 na ADMB4 F 0.88 1.20 2.23 0.81 0.18 0.14 ADMB4 B 0.44 0.51 0.51 0.82 0.13 0.20 NRC ADAPT F 3.45 4.01 8.25 1.46 0.29 0.80 NRC ADAPT S 0.55 0.67 1.21 0.84 0.24 0.79 NRC ADAPT B 0.95 0.89 1.57 0.75 0.26 0.82 ASA S 0.66 0.47 1.09 0.28 0.13 0.22 SS-P3 B 0.82 0.78 1.53 0.38 0.19 0.14 SS-P3 F 2.44 1.86 4.46 0.47 0.23 0.18 SS-P3 S 0.24 0.20 0.37 0.29 0.09 0.09 SS-P6 B 0.24 0.19 0.36 0.30 0.09 0.08 SS-P6 F 0.57 0.34 1.11 0.39 0.17 0.18 SS-P7 B 0.18 0.15 0.27 0.11 0.08 0.11 SS-P7 F 0.52 0.47 0.78 0.32 0.13 0.15 NOTE: Values in table x 100 indicate percentage deviation of estimated values from true values.
|
From page 83... ...
The amount of relative error in this statistic was lower overall than for the previous statistics, especially for model runs using only fishery data (Table 5.4~. Many model runs yielded relative errors well below 50%.
|
From page 84... ...
The ADAPT model generally underestimated these because the natural mortality used was too low. Models using only fishery data (F)
|
From page 85... ...
/true] in Parameters Important for Management Results with Estimated M Model Data sourceEB30 EB~TAC31 TEABC3~RS \DDKF F4.07 3.967.52 0.68-0.56na DDKF S-0.08 -0.510.35 0.46-0.55na ADMB4 F-0.38 -0.200.39 1.24-0.01-0.03 ADMB4 B-0.57 -0.410.09 1.51-0.02-0.10 NRC ADAPT F4.34 4.449.04 0.880.260.23 NRC ADAPT S0.08 0.120.35 0.250.040.21 NRC ADAPT B0.37 0.350.70 0.240.050.21 ASA S0.67 0.340.89 0.130.100.14 SS-P3 B0.04 -0.040.68 0.610.030.01 SS-P3 F2.18 1.414.98 0.880.250.17 SS-P3 S-0.16 -0.220.22 0.450.01-0.01 SS-P6 B-0.13 -0.200.28 0.480.02-0.01 SS-P6 F0.22 -0.070.86 0.530.060.15 SS-P7 B-0.11 -0.13-0.13 -0.03-0.01-0.05 SS-P7 F-0.34 -0.45-0.67 -0.50-0.020.08 TRUE 276 0.08823 0.109309720 NOTE: For the first four management statistics, values within 25% of the truth are shown in boldface type.
|
From page 86... ...
/true] in Important Management Parameters Results with M = 0.225 Model Data source EB30 EB ~TAC31 TEABC3 ~RS DDKF F 5.88 3.16 10.00 0.60 -0.48na DDKF S -0.35 0.29 -0.10 0.39 -0.67na ADMB4 F 0.07 0.66 1.28 1.12 -0.25-0.32 ADMB4 B -0.49 0.02 0.00 0.96 -0.28-0.39 NRC ADAPT F 1.92 2.95 6.37 1.53 0.000.26 NRC ADAPT S -0.05 -0.10 0.27 0.33 -0.190.24 NRC ADAPT B 0.26 0.12 0.62 0.29 -0.180.24 ASA S 0.64 0.84 0.84 0.12 -0.21-0.22 SS-P3 B -0.04 0.17 0.50 0.56 -0.24-0.32 SS-P3 F 4.14 3.63 8.33 0.81 0.13-0.11 SS-P3 S -0.27 -0.13 0.04 0.43 -0.26-0.32 SS-P6 B -0.29 -0.15 0.00 0.41 -0.27-0.32 SS-P6 F 0.15 0.03 0.78 0.55 -0.21-0.13 SS-P7 B -0.36 -0.12 -0.38 -0.04 -0.29-0.37 SS-P7 F -0.54 -0.52 -0.73 -0.41 -0.29-0.22 TRUE 346 0.117 29 0.102 356834 NOTE: For the first four management statistics, values within 25% of the truth are shown in boldface type.
|
From page 87... ...
Yet, it should be noted that no one model performed superbly in all cases and for all management parameters. Effect of M Value The results from these comparisons suggest that having the correct value of M did not significantly improve the assessment results for the ASA and SS models used before and after the true average M value was revealed.
|
From page 88... ...
When the correct average M was used, the NRC ADAPT results were better than the original ADAPT results but still showed major departures from the true values. Therefore, other factors still dominated the assessment results.
|
From page 89... ...
/true] in Important Management Parameters Results With M = 0.225 Model Data source EB30 EB ~TAC31 TEABC3, RS DDKF F 6.34 6.63 8.69 0.32 -0.56na DDKF S 1.71 0.12 3.15 0.53 -0.41na ADMB4 F 0.20 0.21 0.31 0.09 -0.020.01 ADMB4 B -0.07 0.09 0.15 0.24 -0.03-0.08 NRC ADAPT F 7.89 5.94 11.90 0.45 0.201.26 NRC ADAPT S 0.96 0.73 1.61 0.33 0.061.24 NRC ADAPT B 2.16 1.40 2.17 0.00 0.071.25 ASA S 0.77 0.33 1.86 0.62 0.030.27 SS-P3 B 0.15 0.18 0.11 -0.03 0.00-0.02 SS-P3 F 0.38 0.22 0.38 0.00 0.000.10 SS-P3 S 0.00 0.01 -0.13 -0.13 -0.010.00 SS-P6 B -0.06 -0.04 -0.21 -0.16 -0.01-0.01 SS-P6 F -0.14 -0.22 -0.41 -0.32 -0.030.09 SS-P7 B 0.04 0.08 -0.21 -0.24 -0.01-0.02 SS-P7 F -0.09 -0.18 -0.44 -0.38 -0.020.08 TRUE 115 0.024 13 0.132 406780 NOTE: For the first four management statistics, values within 25% of the truth are shown in boldface type.
|
From page 90... ...
/true] in Important Management Parameters Results with M = 0.225 Model Data source EB30 EB ~TAC31 TEABC3, RS DDKF F -0.32 -0.20 -0.23 0.14 -0.88na DDKF S -0.36 -0.17 -0.34 0.04 -0.70na ADMB4 F -0.30 -0.18 -0.06 0.34 -0.08-0.17 ADMB4 B -0.53 -0.36 -0.32 0.45 -0.22-0.32 NRC ADAPT F -0.47 0.22 0.30 1.43 0.200.24 NRC ADAPT S -0.42 1.16 1.09 2.62 0.681.03 NRC ADAPT B -0.42 1.12 1.06 2.54 0.751.15 ASA S 0.17 -0.11 -0.02 -0.16 0.150.27 SS-P3 B -0.30 -0.34 -0.41 -0.16 -0.14-0.18 SS-P3 F -0.30 -0.42 -0.43 -0.18 -0.08-0.07 SS-P3 S -0.17 -0.20 -0.29 -0.13 -0.08-0.11 SS-P6 B -0.11 -0.14 -0.22 -0.13 -0.05-0.07 SS-P6 F 0.17 0.03 0.03 -0.12 0.160.20 SS-P7 B -0.11 -0.13 -0.23 -0.14 -0.05-0.07 SS-P7 F 0.08 -0.01 -0.08 -0.15 0.120.16 TRUE 276 0.088 23 0.109 309720 NOTE: For the first four management statistics, values within 25% of the truth are shown in boldface type.
|
From page 91... ...
Noncoincident but parallel trends of the estimated quantities may be acceptable for stock assessment purposes because the estimated trend is unbiased despite the error in estimation of absolute abundance. That is, even if actual stock abundance values are unknown, it is useful to be able to detect relative increases and decreases of stock abundance over time.
|
From page 92... ...
. The following were used: data set 1 only, no survey information, true average value of M, a logistic function for fishery selectivity, and a plus group starting at age 10.
|
From page 93... ...
Retrospective Analyses As discussed in Chapter 3, retrospective analysis is an essential tool for studying the consistency of stock assessment results and methods over time. Retrospective analyses were performed for the three age-structured methods by a subset of the analysts using the Stock Synthesis model, AD Model Builder, and NRC ADAPT.
|
From page 94... ...
On the other hand, the analysts were provided the true average value of natural mortality rate used to generate the simulated data sets. To illustrate the retrospective results, examples of "good" and "bad" retrospective patterns are given.
|
From page 95... ...
Table 5.15 gives the average absolute relative deviations between estimated and true terminal year exploitable biomass by data set and assessment method or type. When using a criterion of overall average error of 25% or less of the true values as the goal for stock assessment, only 8 of the 50 entries in Table 5.15 (shown in bold)
|
From page 96... ...
The data and models tested the effects of ageing error, variation in natural mortality, changes in fishery selectivity and catchability, a lack of proportionality in the relationship between fishery CPUE and biomass, a change in survey selectivity, a dome-shaped selectivity curve for the survey, underreporting of catch, and random variability in the dynamics and
|
From page 97... ...
These conditions are known to exist in actual stock assessments, and simulation results confirmed that these co-occurring complications can lead to substantial bias and variability in estimates of population and management parameters. Because it is rare to know enough about any specific fish population to estimate parameters for all of the processes that can affect that population, its fisheries, and surveys, it is impossible to know if the five simulated populations could be considered typical of any specific actual population.
|
From page 98... ...
conducted on data set 1 using Stock Synthesis (SS-P3, SS-P6, and SS-P7) , AD Model Builder, and NRC ADAPT; M was fixed at the true average value in all cases.
|
From page 99... ...
, AD Model Builder, and NRC ADAPT; M was fixed at the true average value in all cases.
|
From page 100... ...
conducted on data set 3 using Stock Synthesis (SS-P3, SS-P6, and SS-P7) , AD Model Builder, and NRC ADAPT; M was fixed at the true average value in all cases.
|
From page 101... ...
conducted on data set 4 using Stock Synthesis (SS-P3, SS-P6, and SS-P7) , AD Model Builder, and NRC ADAPT; M was fixed at the true average value in all cases.
|
From page 102... ...
conducted on data set 5 using Stock Synthesis (SS-P3, SS-P6, and SS-P7) , AD Model Builder, and NRC ADAPT; M was fixed at the true average value in all cases.
|
From page 103... ...
TABLE 5.14 Serial Correlation Between Errors in the Estimates of Exploitable Biomassa in the Terminal Year SS-P6 SS-P7 ADMB4 NRC ADAPT Data Set F B S F F B S F B S 1 0.86 0.71 0.56 0.89 0.64 0.54 0.47 0.46 0.38 0.41 2 0.26 0.21 0.46 0.02 0.32 0.51 0.56 0.32 0.27 3 0.66 0.04 0.42 -0.12 0.72 0.42 0.30 0.71 0.66 0.68 4 0.56 0.54 0.39 0.64 0.75 0.22 0.32 0.41 0.40 5 0.39 0.10 -0.04 0.38 0.37 0.07 0.00 0.30 0.08 0.07 aDeviations between estimated and true exploitable biomass (for terminal years 15-30)
|
From page 104... ...
104 a' a' Ed o Cal +1 · _4 ·_4 Cq Cq o .0 a' ·_4 o x Ed o Cq a' .= Cq 3 Cq a' Cq Cq a' Cq Cq ¢ o a' ¢ ¢ Ed E~ ¢ ¢ v z ¢ 1 V, V, V, m V, m V, m 4= V, 4= 0 ~0 0 0 00 ~0 0 0 ~0 00 ~ CM ~0 ~CM ~CM a' a' o o +1 3 Cq Cq o .0 a' o x .~ a' E~ o Cq a' · ~ Cq a' a' · ~ 3 Cq Cq Cq a' Cq Cq ¢ o s~ a' 11 ~ a ~ .
|
From page 105... ...
TABLE 5.19 Percent of Negative Relative Deviations Given a Negative True Change (labeled as a "down" in Exploitable Biomass for Terminal Years 16-30 Assessments SS-P3 SS-P7 ADMB NRC ADAPT Set #Downs F B S F F B F B S 1 9 33% 56% 56% 56% 44% 67% 0% 33% 33% 2 8 25% 63% 50% 50% 25% 100%0% 13% 25% 3 10 10% 0% 10% 0% 20% 10%20% 20% 20% 4 10 30% 30% 30% 20% 10% 50%0% 10% 10% 5 7 57% 29% 57% 57% 100% 100%100% 100% 100% Average 8.8 31% 35% 41% 37% 40% 65%24% 35% 38% NOTE: Change in biomass is between terminal year and previous year. Percent negative deviations below 50% indicate a tendency of the method to over-estimate biomass during periods of population decrease.
|
From page 106... ...
It is obvious that a more comprehensive evaluation of stock assessment methods should be undertaken, given the results of this study. Issues related to the treatment of measurement and process errors, the functional dependence of population
|
From page 107... ...
However, the success of these more complicated models depended on correct specification of the dynamic changes in selectivity, catchability, and natural mortality. Simulation results suggest that models with greater complexity offer promise for improving stock assessment.
|
From page 108... ...
The implication is that calibration of survey catchability is an important consideration; calibration studies should be done when there are changes in vessels, crews, or operations that affect the way a survey is conducted (see ASMFC, 1997~. Retrospective analyses from the simulation revealed that stock assessments can vary substantially from the true values over time.
|
From page 109... ...
To foster excellence in stock assessment, NMFS should continue to support and encourage scientists to engage in creative stock assessment activities (e.g., workshops, gaming sessions, and conferences) so that the process of doing stock assessment does not become routine and stale.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.