Summary

Widely available, trustworthy government statistics are essential for policy makers and program administrators at all levels of government, for private-sector decision makers, for researchers, and for the media and the public. In the United States, 13 principal statistical agencies as well as units and programs in many other agencies produce various key statistics in areas ranging from the science and engineering enterprise to education and economic welfare. Their work is coordinated by the chief statistician in the U.S. Office of Management and Budget (OMB) and the Interagency Council on Statistical Policy (ICSP).

Official statistics are often the result of complex data collection, processing, and estimation methods. These methods can be challenging for agencies to document and for users to understand—they are not intrinsically transparent (a term defined below). The importance of openness of official statistics is noted in the Committee on National Statistics’ Principles and Practices for a Federal Statistical Agency (NASEM, 2021):

Principles and Practices is not the only high-level source calling for transparency in official statistics. Other sources include OMB’s Standards and Guidelines for Statistical Surveys, the Federal Data Strategy, and the Foundations for Evidence-Based Policy Act. These important sources all recognize the importance of transparency and the closely related concept of reproducibility, and they all call for agencies to strive to achieve transparency in their work.

TRANSPARENCY AND REPRODUCIBILITY

The panel could find no formal definition of the term transparency when used in conjunction with official statistics, though its desirability is often cited. Two cases in which the transparency of official statistics is touched upon are the following. First, the Quality Assurance Framework of the European Statistical System Version 2.0 contains this language:

Second, the UN Statistics Quality Assurance Framework, which defines transparency in conjunction with objectivity, impartiality, and professionalism, states that transparency is …

This report has drawn on existing definitions of reproducibility, and defines what transparency and reproducibility mean in this particular context. For our purposes, transparency is the provision of sufficiently detailed documentation of all the processes of producing official estimates. The goal of transparency is to enable consumers of federal statistics to accurately understand and evaluate how estimates are generated. There are different levels of understanding. Since consumers vary in their interests and needs, transparent documentation includes basic information for the merely curious observer as well as technical information for experts. Similarly, there are different levels of evaluation, ranging from individual impressions about the usefulness of official estimates for idiosyncratic purposes to the most rigorous form of assessment, an attempt to reproduce the estimates in an independent investigation. Transparency makes it possible

to understand how official estimates came to be as they are, and whether they are reliable.

While investigations of the reliability of federal statistics by outside organizations are not common, it is essential for agencies to have available the information necessary for them to be undertaken. The credibility of official estimates is undermined if questions about their genesis cannot be answered or if it is impossible to check them. The recent National Academies of Sciences, Engineering, and Medicine report on Reproducibility and Replicability in Science (2019b) observes that there are some general questions about the reliability of research results that have been raised across all scientific disciplines:

1. Are the data and analysis laid out with sufficient transparency and clarity that the results can be checked?
2. If checked, do the data and analysis offered in support of the result in fact support that result?
3. If the data and analysis are shown to support the original result, can the result reported be found again in the specific study context investigated?
4. Finally, can the result reported or the inference drawn be found again in a broader set of study contexts? (p. 44)

In this report, we are concerned with transparency in relation to the first three questions. Answering questions 1 and 2 involves scrutinizing and employing information from the statistical agency to check results that the agency has published. This sort of investigation—notably involving data analysis (including “cleaning,” editing, and weighting) and associated computer code—seeks to determine if the published results based on these stages of the research process can be reproduced. Answering question 3 involves conducting a much broader independent investigation, within the “specific study context” that produced the original official estimates. The context is the full set of study components, from conceptualization to design to data collection to data analysis and publication.

Can conducting a parallel investigation, using the same procedures as those followed by the agency to construct the official estimates, reproduce those estimates, within a reasonable margin of error? Further, one should have a prespecified margin of error that one anticipates from reproducing estimates from independent studies. This idea is specified in reproducibility exercises done by people connected with the Center for Open Science. Recognizing that such studies would be very complex and expensive, our recommendations for transparent documentation are aimed at urging agencies to have available the information needed to make them possible.

Finally, transparency includes discoverability. Data users have to be able to readily locate sufficient information about current and past statistical programs and sets of estimates and input datasets to be both aware of their existence and to evaluate their quality and fitness for use.

Transparency in federal statistics has multiple benefits, as highlighted in Box S-1. First, transparency is consistent with sound management. If federal agencies understand that their internal processes are subject to external scrutiny, this will encourage thoroughness and care. Further, agencies can better cope with the inevitable temporary and permanent changes to staff if they retain, in an accessible way, detailed information as to how they accomplish their various data collection, data treatment, and estimation tasks in the production of official statistics.

Second, transparency facilitates innovation and improvement in statistical methods. Internal researchers from a statistical agency producing official statistics, as well as external researchers, are in a better position to enhance methods for data collection or estimation if clear and detailed documentation of existing processes and sources for producing official estimates is maintained and made accessible.

Third, transparency buttresses confidence in statistical methods. Federal statistics have broad impact: they are used to apportion political representation to states, to form congressional districts, and to allocate substantial governmental funds to regions. They are also used to inform and assess the effectiveness of policies to improve public health, education, the economy, employment, agriculture, and commerce. One critical component of trust—both of building trust and earning it—is for federal statistical agencies to

“open their books” to the extent feasible, that is, to be transparent as to the data they collect and the methods they use to produce the official statistics.

Along the same lines, transparency is closely aligned with the reproducibility of a set of official statistics, whose value for scientific activities has lately acquired renewed importance. Due to the difficulty of controlling precisely what happens during a survey interview, reproducibility—that is, the ability to repeat the entire process of planning and design, data collection, data treatment, and statistical estimation in support of the production of a set of official statistics—is difficult to attain. It therefore can be argued that one should assess reproducibility in this context in the same way it was defined in Reproducibility and Replicability in Science (NASEM, 2019b). The definition in that report focuses on reproducing the computational steps used to produce the findings or results. To be in agreement with that notion of reproducibility, statistical agencies would need to retain the code used for all data treatments and estimation steps. If one then used the archived input datasets as inputs into this retained code (in the appropriate computer environment) and arrived at the identical official statistics, this would support the statement that a given set of official statistics was (computationally) reproducible. However, we believe that this sense of reproducibility is not sufficient. The panel feels that any components of the planning and data collection processes that are repeatable, such as the use of the computer code developed in support of computer-assisted survey techniques, and the use of the survey instrument itself, also need to be documented and retained for review. Therefore, the panel argues for a sense of reproducibility that is more comprehensive than computational reproducibility.

Fourth, transparency increases data utility. Users of information collected by federal agencies have to be able to understand its strengths and limitations. Informed data use leads to sounder conclusions and policy formation. This is true whether data users are creating their own tabulations from a single data collection or combining information from several sources. As federal agencies augment their data collection efforts to include more use of administrative records and other nonsurvey data, transparent processes become even more important.

Recognizing the need to enhance the transparency and reproducibility of its statistics, for all of the above reasons, the National Center for Science and Engineering Statistics (NCSES) asked the National Academies’ Committee on National Statistics to establish a consensus Panel on Transparency and Reproducibility of Federal Statistics to study issues of documentation and archiving of NCSES statistical data products.

In addition, NCSES asked the panel to consider how NCSES could work with other federal statistical agencies to facilitate the adoption of currently available documentation and archiving standards and tools. The use of metadata standards and tools has been widely found to assist in both

the documentation of methods and the archiving of official statistics and the input data used in their creation. In addition, if U.S. federal statistical agencies and various international statistical offices are to benefit from sharing methods, data, and results with each other, such efforts would be eased by making use of common tools for documentation and exchange.

TRANSPARENCY AS A NECESSARY COMPONENT OF FEDERAL STATISTICS

The panel reviewed federal legislation, OMB statistical policy directives and memoranda, best practice documents, including Principles and Practices, and other sources to identify what is variously required and recommended for running an effective federal statistical agency. The panel concluded that an effective agency should embrace transparency by providing important information that demonstrated independence from political and other undue external influence, supporting its consistency with current state-of-the-art methods, and exhibiting respect for and protection of data providers.

In its investigations, the panel came to the conclusion that the agencies of the federal statistical system are reasonably complete in what they retain internally for their agency. This, however, is not as true regarding the archiving of input datasets. Externally, agencies are not as transparent with respect to what is available to the public regarding data treatments and the methodologies employed. Also, agencies often do not provide information to researchers about what access is permitted and available via secure environments regarding input datasets.

It is clear from a wide variety of sources that transparency is an important goal to strive for in federal statistics, and that it is needed regarding the operations used, the methods applied, and the results obtained. There are several benefits to being transparent, but the most important are supporting

greater trust that the official estimates are produced in an unbiased manner and supporting greater trust that they are of high quality.

Given the importance of greater transparency of federal statistics, we offer the following recommendation.

ARCHIVING PRACTICES

Federal law requires agencies to maintain record schedules, to preserve records considered to have long-term value in the National Archives of the United States, and to destroy records lacking value after a specified period.

By doing this, the federal statistical agencies can then leverage their records schedules to improve transparency in their statistical programs.

When such paradata are associated with a statistical program that is used to distribute political power or substantial federal funds (such as the Decennial Census), and the paradata are a key measure of the quality of inputs to such a program, statistical agencies should make public such assessments for relatively disaggregated demographic-geographic domains.

TOOLS TO FACILITATE DOCUMENTATION OF METHODOLOGICAL PROCESSES

There are several tools widely used in academia and in industry that can facilitate the development and documentation of software processes. Since the federal statistical agencies use software processes to help to collect data, treat data in preparation for use in estimation, and estimate a set of official statistics, these tools should be examined for their utility in supporting greater transparency of official statistics.

This last recommendation is important, because transparency of computational processes is as important as the transparency of other processes, and also because it will make their transparency efforts more efficient.

ISSUES CONCERNING TRANSPARENCY AND METADATA STANDARDS

The panel identified several aspects of the use and development of metadata standards relevant to federal statistical agencies. In addition, the federal statistical system as a whole, including NCSES, needs to lead in framing systemwide standards for the documentation and archiving of official statistics. This work should review and incorporate, where appropriate, international standards and processes that are widely in use, and it should provide training for relevant employees.

Such a pilot project could start with programs within agencies and agencies entering into more data sharing and data reuse agreements through the use of current methods, and then proceeding incrementally, assessing how sharing is facilitated through the use of existing metadata standards and tools. When one agency makes some of its data discoverable, this contributes to another agency’s success in being transparent about its own inputs. Therefore, this greater emphasis on data sharing and data reuse is another motivation to make data discoverable.

In addition to such pilot projects, the leadership of the federal statistical system should encourage its agencies to play a more active role in the development of metadata standards and tools.

Finally, the Chief Statistician and the ICSP should develop a continuing education program for agency staff about the nature and purpose of federal government requirements:

RECOMMENDATIONS FOR NCSES

As part of its investigations, the panel reviewed NCSES’s general publication standards as well as information available on NCSES’s Web pages for four programs. These four were chosen because they included both individuals and institutions as respondents, at least one program involved data collected by the Census Bureau, and another involved data collected by a private contractor.¹ In all four cases, the survey design was well described. However, more than one program requested users to go to a technical report to find out what data treatments were used to address nonresponse or failed edits. Further, it was not always clear where the input data or resulting official estimates were archived.

The panel also heard from two expert users of NCSES data, who praised the value of NCSES data but indicated ways in which they could be made more useful for various types of analysis. The panel also learned that NCSES has recently created considerably improved user interfaces for users to prepare customized tables. Based on this, the panel identified several priority areas for NCSES to address so that its data series can be more transparent and helpful to users.

___________________

¹ The four programs were Business Enterprise Research and Development Survey, Survey of Earned Doctorates, Higher Education Research and Development Survey, and Early Career Doctorates Survey.

contact attempts, the interim and final case dispositions, the duration of completed interviews), identify programs that would benefit from the use of paradata, identify what paradata are valuable to maintain, determine the length of time such data can be made available to researchers, and ensure that records schedules include the status of such data. While individual programs have different requirements and uses, for the purpose of transparency NCSES management should develop a policy concerning the availability and use of paradata consistent with its mission.

SYSTEMWIDE BEST PRACTICES

The panel tried to assess the degree to which the information currently provided by NCSES and the other federal statistical agencies about their programs and series of official statistics is consistent with the goals of being transparent. Many agencies do not have formal guidelines or rules that their statistical programs can use to decide what information to provide both internally to agency staff and publicly to their user communities. Instead, these decisions are often made at the programmatic level. Consequently, the panel sent an informal questionnaire to 11 high-profile federal statistical programs, asking appropriate staff to describe their program

documentation and archiving practices. The panel also examined the Web pages associated with these programs.

The panel concluded that in general, agencies make a serious effort to provide information to enable users to understand how estimates were produced, and they include substantial information about the quality of any survey data inputs. However, agencies often do not provide details about the methods used to produce the treated data, and they often limit their published assessments of the quality of the resulting estimates to discussions of sampling error (for survey inputs). Moreover, agencies often do not make it entirely transparent as to where the archived data inputs and outputs are stored and what access to them is possible.

Because documenting and archiving federal statistical series is complicated, the panel identified best practices that agencies can use to determine the extent to which their policies are consistent with transparency. These best practices are divided into tables, in Chapter 7, that provide direction for agencies on what information they should retain internally and what should be made available to the public by establishing documentation requirements for the basic elements of a statistical program (Table 7-1), programs that utilize survey data (Table 7-2), programs that utilize administrative and digital trace data (Table 7-3), issues that arise in data integration (Table 7-4), paradata (Table 7-5), and archiving of data used in the production of official statistics (Table 7-6).

The processes giving rise to the production of official statistics typically involve (1) data collection, often through a survey but sometimes through use of administrative records or digital trace data; (2) data treatments, often to address failed edits or nonresponse; and (3) estimation, the use of methodological procedures to produce the official estimates. Once estimates are produced, (4) their quality needs to be assessed. Based on that division into component processes, the panel provides the following recommendation on what agencies are to retain.

The goal of the agencies is to strive for reproducibility, which in addition to computational reproducibility includes the retention of the documentable portions of the processes used prior to and as part of data collection. So, for example, if a program collected data using computer-assisted survey methods, the computer code used in such techniques would also need to be retained. That is, a record of all processes, even the human components, would be retained since doing so is a basic step in understanding and measuring the reliability of a process, as one might find in quality standards such as the ISO 9000 family² or the Capability Maturity Model Integration.³ Human components could include such things as field protocols, or the treatment of data collected during natural disasters. To this end, the notion of computational reproducibility has to be expanded to include predata collection and data collection processes.

This notion of reproducibility is a more comprehensive notion than transparency, since it involves the actual detailed code used to produce a set of official statistics, rather than, say, only a detailed report about the methodology applied.

___________________

² For more information see https://www.iso.org/iso-9001-quality-management.html.

³https://cmmiinstitute.com/.