Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies (2022) / Chapter Skim
Currently Skimming:
1 Introduction
1 Introduction
Pages 15-32
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 15... ...
Often, these official estimates are used to support decisions about how governmental policies should be implemented or modified to improve various dimensions of the nation's welfare. Further, official statistics are used to allocate political power and to distribute federal funds.
|
From page 16... ...
As indicated, these questions about transparency and reproducibility in official statistics are part of a much broader set of questions about transparency and reproducibility in science. These questions resulted in the National Academies of Sciences, Engineering, and Medicine undertaking a series of forums on open science, whose goal was to elicit from participants a variety of ideas on ways to support greater openness in a wide range of scientific enterprises.
|
From page 17... ...
As with any complicated manufacturing process, for the production of statistics it is important to have a complete workflow history documenting how data are collected, how they are treated, how estimations are carried out, and how the quality of official estimates is assessed. These complete workflow histories allow federal statistical agencies to better manage and innovate in the production of official statistics.
|
From page 18... ...
DEFINITIONS OF TRANSPARENCY AND REPRODUCIBILITY The panel could find no formal definition of the term transparency when used in conjunction with official statistics, though its desirability is often cited. Two cases in which the transparency of official statistics is touched upon are the following: (1)
|
From page 19... ...
… ; ensuring that statistics are deter mined by statistical considerations and not by pressure from providers or users and explaining major changes in methodology to users.3 For our purposes, transparency is the provision of sufficiently detailed documentation of all the processes of producing official estimates. The goal of transparency is to enable consumers of federal statistics to accurately understand and evaluate how estimates are generated.
|
From page 20... ...
Answering question 3 involves conducting a much broader independent investigation, within the specific study context that produced the original official estimates. By "context" here we mean the full set of study components, from conceptualization to design to data collection to data analysis and publication.
|
From page 21... ...
Efficiency arises at an agency producing statistics when what is done to produce them is known so completely so that any temporary or permanent changes to staff due to resignations, retirement, or sickness can be easily accommodated. As part of any organization that undertakes or oversees complex processes, it is necessary for statistical agencies to retain, in an a ccessible way, detailed information as to how they accomplish their various data collection designs, data treatments, and estimation tasks as components of the production of their official statistics.
|
From page 22... ...
Having an accurate and complete description of how an official statistical series is developed gives external users, especially those with some relevant exper tise, confidence that the agency is approaching the data collection and estimation problem with care and objectivity. Along the same lines, knowing which input datasets were used to produce a set of estimates, with the statistical methods used to prepare them for the estimation methodology as well as the details of the estimation methodology, promotes trust.
|
From page 23... ...
In their Policy on Informing Users of Data Quality and Methodology, Statistics Canada outlines its following policy on transparency in data and methods: Statistics Canada, as a professional agency in charge of producing of ficial statistics, has the responsibility to inform users of the concepts and methodology used in collecting, processing and analyzing its data, of the accuracy of these data, and of any other features that affect their quality or ‘fitness for use'….5 In the European Union, Eurostat states that its Code of Practice is the cornerstone of the common quality framework of the European Statistical System … based on 16 Principles covering the institutional envi ronment, statistical processes and statistical outputs…The development, production and dissemination of our statistics are based on sound method ologies, the best international standards and appropriate procedures that are well documented in a transparent manner.6 More recently, in the United States, the Report of the Commission on Evidence-Based Policymaking (2017) 7 -- which the federal statistical agencies are using to guide them in updating their own programs and methods over the next several years -- includes the following statements: • Government also can dramatically improve transparency about its col lection and use of data, improving the American public's ability to hold the government accountable.
|
From page 24... ...
The Federal Data Strategy also urges the adoption of 40 practices, of which eight (strategies 5, 14, 16, 19, 20, 26, 29, and 33) are relevant to this report.8 The Panel wishes readers to see that the views expressed in the reports of these two advisory groups motivate many of the recommendations that are offered here and that implementing these recommendations will result in a more modern federal statistical system, one that uses efficient processes and whose methods and data are more sharable across agencies.
|
From page 25... ...
Although the Census Bureau did retain many component processes and data, failure to retain the entire workflow made it more difficult to evaluate some of those processes, which may have affected preliminary work in designing the 2020 census. Although we did not undertake a formal investigation, we suspect that in the federal statistical system documentation of methods and retention of input datasets is less complete for one-time programs than for continuing programs.
|
From page 26... ...
Statistical agencies have also developed requirements for their internal use to support transparency. An example is the Census Bureau Statistical Quality Standards,12 which contains the following requirements, as summarized in Table 1-2.
|
From page 27... ...
. Survey documentation must be readily accessible to users unless it is necessary to restrict access to protect confidentiality.
|
From page 28... ...
Guideline 7.3.4: Retain all survey documentation according to appropriate Federal records disposition and archival policy. Standard 7.4: Agencies that release microdata to the public must include Documentation and documentation clearly describing how the information is constructed Release of Public-Use and provide the metadata necessary for users to access and manipulate Microdata the data (See also Standard 1.2)
|
From page 29... ...
TABLE 1-2 U.S. Census Bureau's Statistical Quality Standard F2: Providing Documentation to Support Transparency in Information Products Statistical Quality Standard Documentation Requirement F2-1 Documentation that would breach the confidentiality of protected information or administratively restricted information or that would violate data-use agreements with other agencies must not be released.
|
From page 30... ...
For example, the release of some input datasets from surveys could make confidential personal information for individuals or businesses available, which would violate the law as well as undermine trust in the statistical agency. Further, because statistical agencies are increasingly relying on nonsurvey data, including administrative, commercial, and other digital trace data, the data may come with additional restrictions on transparency.
|
From page 31... ...
Rather, it should be understood as an active effort, a way of making information available to a user community so its members can know how a set of official statistics were produced and what their resulting quality is. In this communication, both the audience (the user community)
|
From page 32... ...
Finally, Chapter 7 provides a discussion of best practices for federal statistical agencies. This chapter lays out aspects of what a more modern approach to federal statistics would consist of, including better documentation and archiving practices that provide for data and methodology sharing among agencies and a smoother interface with the public.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.