Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 319 SSA's computations are disaggregated by marital status instead of by race, as done by the Census Bureau. The assumptions regarding fertility, mortality, and immigration differ only slightly. Table 3 presents ∆r values for SSA's projections made since 1950. Whenever the SSA does not give a single median population projection, an average of the optimistic and pessimistic projections is used. Early projections by the SSA published only quinquennial figures, whereas the later projections are for single years. Hence, the table is considerably less dense prior to 1980. As with the Census Bureau's projections, projection errors have become smaller over time. Indeed, since 1980 the SSA has projected the actual population growth rate within 0.08 of a percentage point, but in virtually every instance the projected growth rate has been low. Overall, in treating each value in Table 3 as a separate projection error, the average error is −0.07, indicating a slight tendency to underestimate population growth rates. For both the Census Bureau and the SSA projections, there is a common pattern of errors. Errors tend to be positive for projections that were made between 1960 and 1970 and negative outside that period. The average errors calculated for each column in Tables 2 and 3 are plotted by year in Figure 5. Projections made prior to the mid-1950s are generally too low, because forecasters did not anticipate the baby boom. Between 1960 and 1970 both agencies overshot the population growth, because they failed to predict the baby bust. Fertility fluctuations have been much less pronounced since 1970, and errors have been small overall. Nevertheless, in the latter period there has been a slight tendency for both agencies to underestimate future population growth. CONFIDENCE INTERVALS The previous discussion has focused on median projection series, under the assumption that most users treat them as forecasts. Of course, it is not surprising that median projections have not predicted the future population exactly. In fact, no one can forecast the population exactly except by extreme coincidence. For this reason it is important to place limits within which the future population can be expected to lie. If the limits are extremely wide, they will certainly encompass the subsequent population but will be completely uninformative. On the other hand, very narrow limits will rarely be correct given the uncertainties of future population growth. There are several approaches to creating bounds for population size, three of which are discussed here: (1) high and low projections using alternative “reasonable” assumptions, (2) statistical methods based on the past variability of vital rates, and (3) evaluation of the accuracy of population projections made in the past. Most of the attention below is given to the third approach. The Census Bureau, the SSA, and the United Nations all provide high and low projections that are intended to bracket the expected future population. One 

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 320 TABLE 3 Values of ∆r for SSA Projections of Total U.S. Population Size for Base Years 1950 to 1986 (percent) Duration Base (years) Year 1950 1955 1965 1973 1977 1979 1981 1982 1983 1984 1985 1986 1 −0.0 0.02 −0.0 −0.0 −0.0 −0.0 −0.0 9 6 7 5 3 5 2 −0.0 −0.0 −0.0 −0.0 −0.0 −0.0 6 5 5 7 5 6 3 −0.1 −0.0 −0.0 −0.0 −0.0 6 3 5 7 6 4 −0.0 −0.0 −0.0 4 5 8 5 −0.2 0.11 −0.0 −0.0 5 4 6 6 −0.0 −0.0 7 5 7 −0.0 6 8 −0.1 5 10 −0.6 0.25 1 12 −0.0 5 15 −0.0 0.31 2 20 −0.5 6 25 0.15 30 −0.4 0 Average −0.5 −0.0 0.22 −0.0 −0.1 −0.0 −0.0 −0.0 −0.0 −0.0 −0.0 −0.0 2 4 5 5 8 3 5 7 5 4 5 NOTE: ∆r is defined as where r is the average population growth rate (see text). SOURCE: Projections taken from Myers and Rasor (1952), Greville (1957), Bayo (1966), Bayo and McCay (1974), Bayo, Shiman, and Sobus (1978), Faber and Wilkin (1981), Wilkin (1983), and Wade (1984, 1985, 1986, 1987, 1988). Population estimates taken from data provided by Alice Wade, Social Security Administration, Office of the Actuary. 

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 321 problem with these figures is that the degree of confidence with which they are held is unknown, and there is often considerable controversy surrounding their choice. Indeed, the Census Bureau and the SSA have often been criticized for assuming mortality declines that are too slow or fertility fluctuations that are too conservative (Bouvier, 1990; Crimmins, 1983, 1984; Olshansky, 1988; Ahlburg and Vaupel, 1990). Should users expect the true population outcome to fall within the high and low ranges 25 percent of the time? 75 percent? 99.9 percent? Moreover, because the choice of values for the input variables is somewhat subjective, it is impossible to assign each a specific probability. It would be helpful, however, to have a rough idea of the confidence level thought of by the forecaster.11 For example, in the most recent SSA projections, male life expectancy in 2080 ranges from 75.3 to 82.3 years. Are these values outer bounds of what is possible or rather plausible alternative scenarios, with more extreme ones not out of the question? Keyfitz (1981:590–591) sums up the difficulty as follows: FIGURE 5 Average errors in projected growth rates of U.S. population by Bureau of the Census and Social Security Administration projections. NOTE: See text for definition of ∆r. SOURCE: Data from Tables 2 and 3. Without some probability statement, high and low estimates are useless to indicate in what degree one can rely on the medium figure, or when one 11 Even if precise probability distributions were assigned to each input, one would still question how likely a particular combination of those inputs is. The Census Bureau's high and low scenarios assume perfect correlation of high fertility, low mortality, and high immigration. In reality these inputs are not perfectly correlated. 

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 322 ought to use the low or the high. Nor do we derive any help from the notion that each of the projections corresponds to a different set of assumptions and that it is up to the user to consider the three sets of assumptions, decide which is the most realistic, and choose that one. If he actually goes to the trouble and has the skill to reflect on the alternative sets of assumptions and decide which is most realistic, then he might as well make the calculations in addition—that is a relatively easy matter once the assumptions are specified. If on the other hand, as more commonly happens, the user looks at the results and takes whichever of the three projections seems to him most likely, then the demographer has done nothing for him at all—the user who is required to choose on the basis of which of the results looks best might as well choose among a set of random numbers. The second way of considering the variability in future populations is through the development of complicated statistical models that account for variability in past vital rates (Heyde and Cohen, 1985; Cohen, 1986; Sykes, 1969; Lee, 1974; Lee and Carter, 1990; Saboia, 1974, 1977; Voss et al., 1981a; Alho, 1984, 1985, 1990; Alho and Spencer, 1985, 1990a, 1990b). This approach is typically of the sort that simply extrapolates the rates of the past into the future and accounts for how variable the rates have been. Similarly, Keyfitz (1989) uses the variability of historical vital rates, but works with simulations in place of statistical models. Alho (1985) has developed an approach along these lines that takes into account expert opinion as well. This “mixed” model takes a weighted average of expert predictions and information concerning past trends and their variability to develop confidence intervals about the future population. In one such model he determines that the Census Bureau's high and low projections of births are well within a two-thirds confidence interval—that is, the actual number of births would be likely to fall between the high and the low estimates less than two-thirds of the time. For mortality, however, the high and the low estimates appear to match rather closely a 95 percent confidence interval for short-term projections, with confidence greater than 95 percent for projections longer than 5 or 10 years. A third approach is simply to evaluate the performance of past projections. With a sufficient number of projections, it is possible to view the distribution of errors by size and sign. Smith (1987) shows that the distribution of percent errors is approximately a normal bell-shaped function, implying that most errors cluster quite close to a mean value and become less frequent as one moves away from this mean. Values of ∆r likewise appear to be approximately normally distributed. Many authors have taken advantage of this fact to make predictions about the percentage of future errors that will fall within a given range (Stoto and Schrier, 1982; Voss et al., 1981b; Smith, 1987; Stoto, 1983; Keyfitz, 1981; Long, 1987). It is generally true that an error will fall within a single standard deviation of the mean two-thirds of the time and will fall within two standard deviations of the mean 95 percent of the time. In this way it is sufficient to 

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 323 calculate the standard deviation, typically by using the root mean square error, as defined earlier, to describe the confidence in a projection. This approach has been taken by several authors in considering projections of state or county populations. A few findings are noteworthy. First, projections tend to be much more accurate (lower ∆r values) in areas where growth is moderate than where it is exceptionally slow or rapid (Smith, 1984). Accuracy is considerably better for large states than for small ones (Smith, 1987). Keyfitz (1981) observed that the root mean square error for states is more than double that of the whole country, most likely because interstate migration is much more variable than international migration. After making statistical adjustments for the overall biases in each projection year, Stoto (1983) puts the standard deviation of ∆r for the Census Bureau's projections through 1970 at 0.52 percentage point. Because there are so few years on which this estimate is based, he believes that a better estimate of the standard deviation can be obtained from U.N. projections for developed regions. Using these, the standard deviation is calculated as 0.28. Thus, on the basis of past projection performance, projected future growth rates can be expected to be within plus or minus 0.3 percentage point of the actual growth rates two-thirds of the time.12 Stoto points out that this range of growth rates matches fairly closely the Census Bureau's (1977) high and low projections. One can be 95 percent confident that the true growth rate will fall within plus or minus 0.6 percentage point of the projected rate. After examining the United Nations projections for developed and developing regions, Keyfitz (1981) concluded that the high and low projections are rather close to a two-thirds confidence interval.13 The approach taken by Stoto and Keyfitz assumes that the distribution of errors in the future will mirror that of the past. Smith and Sincich (1988) provide support for this assumption in their analysis of simple projections of county populations, but Beaumont and Isserman (1987) take issue with their methodology. Beaumont and Isserman's (1987:1007) evaluation of state population projections leads to the opposite conclusion, and they believe the following: 12 One might object that the United Nations projections are not the same as those of the Census Bureau and indeed do not even project the same populations. However, if the primary source of variability in r values stems from the inherent volatility of population growth rather than from peculiarities of the projecting agency, use of data from other developed countries may be quite instructive. 13 The general congruence of two-thirds confidence intervals with the Census Bureau's high-low projections apparently holds only with regard to total population size. Alho and Spencer (1985) point out that, when the population is disaggregated into age groups, different patterns can be seen. They demonstrate that for surviving ages (in which persons are already born at the time of the projection) the high-low estimates are close to 90 percent confidence levels. However, part of the error they consider incorporates uncertainty in the baseline population, so confidence regarding the mortality assumptions may in fact be higher than 90 percent. They find that the high-low interval for births is considerably narrower than a two-thirds confidence interval. 

EVALUATING THE ACCURACY OF U.S. POPULATION PROJECTION MODELS 324 The results indicate that the error distributions for the state data vary considerably according to historical context, method, and base period. There does not appear to be a predictable pattern as to when the error distribution for one period might serve as an accurate guide to the error distribution in some future period. It has been noted (see Figure 4) that errors in the Census Bureau's projections have declined over time, most likely because of the greater stability of fertility rates more recently. Following an examination of United Nations projections made between 1958 and 1968, Keyfitz (1981) also observed a reduction in the magnitude of errors over time. In addition to the reasons given by Long (1987) for this decline, Keyfitz suggests that data on population levels and rates of increase have improved. He argues that this trend can be viewed in any of three ways: (1) presume it will continue, so that future projections can be expected to be better than past ones; (2) take the most recently observed variability as correct; or (3) presume that the future will be as variable as the past with apparent trends representing only random variations. A strong case can be made for placing confidence intervals around population projections. Users need to know how likely alternative scenarios of growth may be. Keyfitz (1972) offers as an example a town that is about to embark on building a reservoir. Overestimating the size of the future population incurs a small cost in having built the reservoir too large, but underestimating it entails construction of a costly additional reservoir. In practice the planner takes the largest projection, but this may involve extremely inefficient use of funds. A planner who knows the distribution of possible population sizes could choose the population estimate that minimizes expected losses on a project and plan accordingly. As Keyfitz (1972:360) concludes: The demographer should be encouraged to [give] subjective probabilities by the thought that society bets on future population whenever it builds a school, a factory, or a road. Real wagers, running to billions of dollars, are implicit in each year's capital investment. Someone ought to be willing to make imaginary wagers if so doing will more precisely describe the distribution of future population and so improve investment performance even by a minute fraction. Despite the need for confidence intervals and probability distributions associated with population projections, ma ny problems remain with the methods considered here. (Many methods now exist for estimating the uncertainty surrounding population projections, but seldom has one method been tested against others; see Lee [1974] and Cohen [1986] for exceptions.) As noted in the latest Census Bureau projections: “There is considerable controversy over the means of handling improvements in methods, changing variability in population growth rates, and other complicating factors” (Spencer, 1989:14). The bureau presents bounds using a “reasonable high” and a “reasonable low.”