Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies (2022) / Chapter Skim
Currently Skimming:
Pages 12-25
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 12... ...
SYSTEMWIDE BEST PRACTICES The panel tried to assess the degree to which the information currently provided by NCSES and the other federal statistical agencies about their programs and series of official statistics is consistent with the goals of being transparent. Many agencies do not have formal guidelines or rules that their statistical programs can use to decide what information to provide both internally to agency staff and publicly to their user communities.
|
From page 13... ...
Recommendation 7.1: The National Center for Science and Engineer ing Statistics (NCSES) and all agencies that produce federal statistics should, to the fullest extent feasible, document their data collection methods, their data treatments, their estimation methodologies, and assessments of the quality of their official estimates, and they should archive their input datasets and their official estimates to support reproducibility and later reuse, as specified in the tables developed by the panel.
|
From page 14... ...
This notion of reproducibility is a more comprehensive notion than transparency, since it involves the actual detailed code used to produce a set of official statistics, rather than, say, only a detailed report about the methodology applied. Recommendation 7.2: Senior management at the agencies that produce federal statistics should provide resources and staff support to help transform their current processes to incorporate the use of data sharing and reuse through use of metadata tools and standards.
|
From page 15... ...
To provide information on these important matters, the statistical agencies produce official statistics on the status of these various aspects of the United States. These estimates must be seen as trustworthy.
|
From page 16... ...
, one attendee, John Gawalt, then director of the National Center for Science and Engineering Statistics (NCSES) , thought it would be valuable to have a study examining the extent to which NCSES practices, and those of federal statistical agencies more broadly, are currently transparent, what benefits greater transparency might offer, what tools might facilitate greater degrees of transparency, what are the legal, administrative, and resource-based constraints on being more transparent, and, if considered desirable, what are the appropriate steps to increase the degree of transparency both in the near term and over time.
|
From page 17... ...
As with any complicated manufacturing process, for the production of statistics it is important to have a complete workflow history documenting how data are collected, how they are treated, how estimations are carried out, and how the quality of official estimates is assessed. These complete workflow histories allow federal statistical agencies to better manage and innovate in the production of official statistics.
|
From page 18... ...
8. How can NCSES work with other federal statistical agencies to facilitate adop tion of documentation and archiving standards and tools in common?
|
From page 19... ...
Similarly, there are different levels of evaluation, ranging from individual impressions about the usefulness of official estimates for idiosyncratic purposes to the most rigorous form of assessment, which is the attempt to reproduce the estimates in an independent investigation. Transparency makes it possible to understand how official estimates came to be as they are, and whether they are reliable.
|
From page 20... ...
This idea is specified in reproducibility exercises done by people connected with the Center for Open Science.4 Recognizing that such studies would be very complex and expensive, our recommendations for transparent documentation are aimed at urging agencies to have available the information needed to make them possible. In a study of the reproducibility of a set of official statistics that was derived from a sample survey, one would use the same definition of the target population (e.g., "adults 18-75 in the United States, living in noninstitutional quarters [not prisons, nursing homes, etc.]
|
From page 21... ...
Efficiency arises at an agency producing statistics when what is done to produce them is known so completely so that any temporary or permanent changes to staff due to resignations, retirement, or sickness can be easily accommodated. As part of any organization that undertakes or oversees complex processes, it is necessary for statistical agencies to retain, in an a ccessible way, detailed information as to how they accomplish their various data collection designs, data treatments, and estimation tasks as components of the production of their official statistics.
|
From page 22... ...
Most importantly, the public must have confidence that all of this is the case. One component -- of both building trust and earning it -- is for federal statistical agencies to "open their books" to the extent feasible.
|
From page 23... ...
In their Policy on Informing Users of Data Quality and Methodology, Statistics Canada outlines its following policy on transparency in data and methods: Statistics Canada, as a professional agency in charge of producing of ficial statistics, has the responsibility to inform users of the concepts and methodology used in collecting, processing and analyzing its data, of the accuracy of these data, and of any other features that affect their quality or ‘fitness for use'….5 In the European Union, Eurostat states that its Code of Practice is the cornerstone of the common quality framework of the European Statistical System … based on 16 Principles covering the institutional envi ronment, statistical processes and statistical outputs…The development, production and dissemination of our statistics are based on sound method ologies, the best international standards and appropriate procedures that are well documented in a transparent manner.6 More recently, in the United States, the Report of the Commission on Evidence-Based Policymaking (2017) 7 -- which the federal statistical agencies are using to guide them in updating their own programs and methods over the next several years -- includes the following statements: • Government also can dramatically improve transparency about its col lection and use of data, improving the American public's ability to hold the government accountable.
|
From page 24... ...
This memorandum follows other recent official memos emphasizing the need for better record retention (at the National Archives and Records Administration) , especially administrative records data, in order to support greater reuse of collected data both by researchers and by other statistical agencies.
|
From page 25... ...
Although the Census Bureau did retain many component processes and data, failure to retain the entire workflow made it more difficult to evaluate some of those processes, which may have affected preliminary work in designing the 2020 census. Although we did not undertake a formal investigation, we suspect that in the federal statistical system documentation of methods and retention of input datasets is less complete for one-time programs than for continuing programs.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.