Practice 10

Openness About Sources and Limitations of the Data Provided

A statistical agency must be transparent about how it acquires data and produces statistics and be open about the strengths and limitations of its data. No matter how high-quality the statistical data are, they will contain some uncertainty and error. This does not mean the data are wrong but that they need to be used with an understanding of their limitations. Statistical agencies need to communicate clearly to a wide range of potential users what the uncertainty in the data means for using the statistical information appropriately.

To be most effective, openness should be tailored to different user groups. For press releases disseminated to the public, the agency must make every effort to note both the meaning of the statistics and their limitations for various uses. For more technically trained users, detailed descriptions of methods and measures of quality should be made available (see Federal Committee on Statistical Methodology, 2001).

Openness requires that statistical releases from an agency include a full description of the purpose of the program; the methods and assumptions used for data collection, processing, and estimation; information about the quality and relevance of the data; analysis methods used; and the results of research on the methods and data (see NASEM, 2019c). Such transparency is essential for credibility with data users and stakeholders and for public trust. Thus, openness about statistical limitations requires much more than providing estimates of sampling error. In addition to a discussion of nonsampling errors (e.g., coverage errors, nonresponse errors, measurement errors, and processing errors), it is valuable to have

a description of the concepts measured and how they relate to the major uses of the data. Descriptions of the shortcomings of the data should be provided in sufficient detail to permit a user to take them into account in analysis and interpretation. Descriptions of how the data relate to similar data collected by other agencies should also be provided, particularly when the estimates from two or more surveys or other data sources exhibit large differences that may have policy implications.

There is often a tension between timeliness and accuracy. When concerns for timeliness prompt the release of preliminary estimates (as is done for some economic indicators and has been done in response to COVID-19), consideration should be given to the frequency of revisions and the mode of presentation from the point of view of the users as well as the issuers of the data. Agencies that release preliminary estimates must educate the public about differences among preliminary, revised, and final estimates.

An important aspect of openness concerns the treatment of errors that are discovered subsequent to data release. Openness means that an agency has an obligation to issue corrections publicly and in a timely manner. The agency should use not only the same dissemination avenues to announce corrections that it used to release the original statistics, but also additional vehicles, as appropriate, to alert the widest possible audience of current and future users of the corrections in the information. Agencies should be proactive in seeking ways to alert potential users of the data about the nature of a problem and the corrective actions that it is taking or that users should take.

Federal statistical agencies should implement quality frameworks for their programs and use these to describe the strengths and limitations of the statistical information produced by the data (see Practices 3 and 6). Some statistical agencies have developed “quality profiles” for major surveys, which document what is and is not known about errors in estimates to help users.

As agencies use administrative data and other alternative sources, they need to provide information not only on what is known about those sources but also on how the data were linked or blended with other data sources and the potential errors introduced through linkage. This is a challenging and evolving area (see NASEM, 2017b, 2017d), and efforts are currently underway to develop best practices and quality frameworks

for these topics (see e.g., Czajka and Strange, 2018; Federal Committee on Statistical Methodology, 2019, 2020).

Statistical agencies should treat the effort to provide information on the quality, limitations, and appropriate use of their data as an essential part of their mission. Such information and metadata should be readily accessible to all known and potential users (see NRC, 1993a, 1997b, 2007b). By being open about the sources and limitations of their data and by providing as much data as possible in ways that are as easy as possible for users to access and apply, federal statistical agencies fulfill their vital mission to inform the public, contribute to evidence-based policy making, and support the development of societal knowledge for the public good.

This page intentionally left blank.