State and Local Government Statistics at a Crossroads (2007) / Chapter Skim
Currently Skimming:
4 Data Quality and Statistical Methods
4 Data Quality and Statistical Methods
Pages 69-93
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 69... ...
After a brief discussion of dimensions of quality, this chapter considers several issues of statistical methodology and reporting that affect the quality and usability of the data from the Census of Governments and annual and quarterly surveys of government finances and employment: sample frame development and design, data collection, unit nonresponse, editing and imputation, estimation, data processing, revision policies, and cognitive testing of questionnaires. For each of these topics, the panel makes recommendations for the Governments Division's research and development program.
|
From page 70... ...
The special case of the Quarterly Tax Survey, which has been undergoing a major redesign while the panel examined the state and local government statistics programs, is discussed separately. The accuracy of data from a census or a survey begins with the development of concepts, methods, and design; continues with the necessary steps of data collection; and includes processing and editing, the development of estimates, and data analysis.
|
From page 71... ...
The list of state and local governments is maintained in the Governments Integrated Directory. This list is updated periodically with information on newly established government units that meet Census Bureau definitions, and dissolved or inactive units.
|
From page 72... ...
Special districts present a problem for sample selection because many of them have existed for relatively brief periods and have wide geographic boundaries, which can cross state lines. The best opportunity to address these concerns is during the redesign of the Quarterly Tax Survey, discussed later in this chapter, and the panel encourages the Census Bureau to take full advantage of this and similar opportunities to assess the quality of the sample frame for the survey.
|
From page 73... ...
cThese local governments account for about 65 percent of all local nonproperty tax collections. dThe 100 largest systems account for about 85 percent of all national activity.
|
From page 74... ...
The Annual State and Local Government Public Employee Retirement System Survey was converted to a probability basis in 2004. However, nonprobability methods continue to be used to select the sample for the Quarterly Public Employee Retirement System Survey and a portion of the Quarterly Tax Survey.
|
From page 75... ...
As an example, the division now operates three separate collection systems for gathering the periodic Census of Governments and ongoing survey data: a division-managed mailout/mail-back questionnaire for state and local governmental units in 22 states and the District of Columbia; direct collection from 48 large units, which is also managed by the division; and a central collection program, managed by the states, under which data on the state's localities are collected and transmitted to the division by 28 states. To complicate matters further, in some cases the data are provided on paper; in other cases, the data are provided in standardized electronic formats; and in still other cases, the data are provided in nonstandardized electronic formats that are negotiated between the reporters and the Governments Division.
|
From page 76... ...
Table 4-2 shows the response rates for noncentral collection states for the 2004 Annual Finance Survey: the median rate for general governments (counties, cities, etc.) is 79 percent, while the median rate for special districts is only 62 percent.
|
From page 77... ...
This is one reason for the delay in the publication of estimates from the surveys. The Governments Division has not conducted a rigorous study of the impact on data quality of state central collection for the Annual Finance Survey.
|
From page 78... ...
While the Governments Division regularly publishes unit response rates to its surveys, these typically include only unweighted rates. An exception is the 2005 Annual Survey of Public Employee Retirement Systems, for which the unweighted response rate among eligible governments was 92.5 percent, but the weighted response rate, based on the value of holdings and invest
|
From page 79... ...
While the Governments Division routinely publishes unit response rates, it has not historically calculated item nonresponse rates, except for the portion that is attributable to unit nonresponse and which is not ordinarily counted as part of item nonresponse. In part, this practice reflects the ambiguous nature of nonresponse in some cases.
|
From page 80... ...
Conclusion 4-2: The documentation of nonresponse, particularly item nonresponse, in Governments Division surveys is inadequate to inform users or to help plan effective means to increase response and improve accuracy of the data. Recommendation 4-3: The Governments Division should provide more complete documentation of unit and especially item nonresponse for its surveys of state and local governments.
|
From page 81... ...
The Governments Division, however, has not systematically examined the extent of bias that may be caused by unit or item nonresponse, whether nonresponders are similar to responders, and whether adjustments can be made using statistical or other techniques to correct for the biases. The availability of other sources of data for many state and local governments should make it easier for the division to study nonresponse bias than is often the case.
|
From page 82... ...
Neither of these standard practices is followed in compensating for nonresponse to the major Governments Division surveys. The form of imputation used for item nonresponse in the Quarterly Tax Survey -- namely, carrying forward previously reported values -- generates a bias in that it dampens change in the data series.
|
From page 83... ...
If statistically based imputation were found to be as effective in correcting for unit nonresponse among nonself-representing units, this could free up resources to pursue quality improvements in other areas. Likewise, by all accounts, the currently used editing procedures for item nonresponse produce substantial added value, but too little is known about how these edits affect overall data quality.
|
From page 84... ...
ESTIMATION Estimates for the annual and quarterly surveys are produced for a variety of domains -- state totals and subtotals of local governments as well as national totals, depending on the survey. Two issues for estimation are the effectiveness of regression adjustments of direct sample estimates and providing information to calculate the precision of estimates of change within and across sample design periods. A domain may also refer to a particular type of government function rather than all functions combined.
|
From page 85... ...
should eval uate the effectiveness of a model-based approach or other method of borrowing strength in yielding improved estimates for small do mains from state and local government surveys. Overall, the appli cation of regression-based adjustments to direct sample estimates should be reviewed to determine which adjustments produce the most improvement.
|
From page 86... ...
StEPS is a generalized processing system that is now used by other economic surveys at the Census Bureau. It has standard data set structures and modules that perform administrative functions, post-collection processes, and support for data collection technologies (Ahmed and Tasky, 1999)
|
From page 87... ...
REVISION POLICIES Because estimation is improved with the benefit of hindsight through the collection of additional data, the Governments Division revises its data series when new data are collected. With respect to all annual and quarterly series (except for the Quarterly Tax Survey)
|
From page 88... ...
4-1 The fact that all 12 consecutive revisions were upward revisions also suggests that there may be a downward bias in the original estimates. Such a bias could result from the fact that the Governments Divisions fills in missing items by carrying over reported items for the same government from previous quarters.
|
From page 89... ...
The redesign of the 2005 version of the Annual Finance Survey is the only example in the Governments Division domain of the use of modern prefield cognitive design techniques. The cognitive testing was a cooperative effort between the Governments Division experts and the survey design staff of the ESMPD.
|
From page 90... ...
The Quarterly Tax Survey now consists of three pieces: universal coverage of state government taxes, a probability sample of county areas to obtain county property tax collections, and a nonprobability sample of local government nonproperty taxes. While the property tax sample is drawn at the county level, data must be collected in each county for all local areas with independent taxing authority.
|
From page 91... ...
The Governments Division has never cal culated coefficients of variation for its estimates of local property taxes, nonproperty taxes, or total tax revenue. This is contrary to good statistical practice and violates Census Bureau standards, so a system must be designed to compute these coefficients in conjunc tion with the sample redesign.
|
From page 92... ...
In view of the shortage of resources, it may be advisable for the Governments Division to start out by integrating the testing and development of new methods and procedures into the upcoming redesign of the Quarterly Tax Survey, to which it is already committed. Recommendation 4-11: The Governments Division should use the re design of the Quarterly Tax Survey to assess the quality of the sample frame, to develop a probability sample of local governments for non property tax measurement, to streamline questionnaires, and to develop cost-effective variance estimation, editing, and imputation procedures that meet Census Bureau standards.
|
From page 93... ...
Scheduled survey redesigns and recurring post–Census of Governments sample updating operations can be used as test beds for improving practices and procedures that can be widely applied across the various data collections. Outside advice and guidance can be obtained from advisory structures, such as the advisory group discussed in Chapter 6, and by regularly scheduling sessions with the American Statistical Association component of the Census Advisory Committee of Professional Associations.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.