Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies (2022) / Chapter Skim
Currently Skimming:
Pages 133-166
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 133... ...
MAKING THE PRACTICES OF NCSES MORE TRANSPARENT 133 TABLE 6-1 Continued Who Collects Name the Data / (Acronym) Type Where Documentation is Available Survey of Westat/ https://www.nsf.gov/statistics/srvydoctoratework/; Doctorate https://www.nsf.gov/statistics/srvydoctoratework/#sd; Recipients sample (SDR)
|
From page 134... ...
NOTE: The National Training, Education, and Workforce Survey (NTEWS) is a new NCSES survey currently under development with plans for the survey to be performed under a negotiated Interagency Agreement with the Census Bureau.
|
From page 135... ...
can be ac cessed by researchers who apply to the Census Bureau for microdata access through federal statistical research data centers (FSRDCs)
|
From page 136... ...
NCSES employees who work in the SSDC must have Special Sworn S tatus with the Census Bureau. Any output that is removed from the SSDC, even to be shared with other NCSES employees, must go through Census disclosure avoidance review to ensure that no information protected under Title 13 or Title 26 protected data is disclosed.
|
From page 137... ...
Publication Standards Utilized By NCSES NCSES currently operates in accordance with its publication standards, as represented in their internal (not publicly available) document, "Statistical Standards for NCSES Publications." Below is the current list of standards regarding documentation of data treatments, methods, and dissemination.
|
From page 138... ...
TRANSPARENCY FOR EXTERNAL USERS OF NCSES SURVEY OUTPUT To assess NCSES's transparency, the panel was interested in the extent to which NCSES provided information on its own Website, over and above its publication standards, to inform NCSES's external user community about various details concerning its statistical programs. Such details could include information about survey designs, the survey instruments used to collect responses, details about how to instruct the field interviewers, the extent of nonresponse and failed edits, how the survey weights were computed, the estimation methodology used, and the variability of those estimates.
|
From page 139... ...
First, for SED data from 1958 onward there is an interactive data tool that can create custom tables of the number of doctorate recipients by demographics, discipline, and institutional characteristics. Researchers can also access SED microdata through the Secure Data Access Facility and the FSRDC.
|
From page 140... ...
The questionnaire is available on the NCSES ECDS Web page. Data treatments include editing and logical and hot-deck imputation for item nonresponse.
|
From page 141... ...
EASE-OF-USE OF INFORMATION FOR ANALYSIS PURPOSES The panel was also interested in ease of access to and use of the infor mation provided, especially the ease of analysis of the official estimates themselves. Therefore, the panel asked two expert users of NCSES data for their views on the uses of the official estimates provided and associated information on issues such as the quality of the data.
|
From page 142... ...
The first task is to examine the information that NCSES provides about its programs both internally and externally, especially the input data and official statistics and
|
From page 143... ...
Second, as expressed in our statement of task, the panel has been asked to address many of the same questions for the entire federal statistical system that it has been asked to address for NCSES. As a result, in Chapter 7, as well as in previous chapters, we present a number of recommendations for ways the federal statistical agencies can more comprehensively document methods and archive their official statistics and input data.
|
From page 144... ...
from the program's Website, to the extent possible consistent with confidentiality protections. As is true for most of the federal statistical agencies, NCSES has made only modest use of shared metadata standards, in particular in its data documentation and in its exchange of official statistics and the associated methods.
|
From page 145... ...
NCSES should • Establish ongoing data user groups with contact mechanisms; • Establish a repeated survey of users as to their current experi ences in accessing and using agency data and how estimates could be presented to facilitate time series and cross-sectional analyses; • Ensure consultations with data users prior to making changes in dissemination systems, statistical programs, and time series; • Create a mechanism that enables members of a statistical pro gram's user group to communicate directly with one another;
|
From page 146... ...
146 TRANSPARENCY IN STATISTICAL INFORMATION • Organize regular meetings with broad user community represen tation; and • Through surveys and direct interactions with users, identify ways to improve the transparency, accessibility, and usability of N CSES estimates, data products, documentation, and dis semination systems, including the structure and navigation of the agency's Website.
|
From page 147... ...
or Interagency Council on Statistical Policy should consider monitoring how closely the principal U.S. federal statistical agencies follow these tables, acknowledging those agencies that come close to complete adherence to them.
|
From page 148... ...
In addition, for issues such as what to retain regarding administrative record data sources, use of digital trace data, and modelbased estimates, we believe that the guidelines we submit are a reasonable start for documenting research in still developing areas. 1 https://www.aapor.org/Standards-Ethics/AAPOR-Code-of-Ethics.aspx.
|
From page 149... ...
the nature of the products, e.g., tabulations, confidential, microdata, or public-use files. Justification for the statistical Description should Information should program and input data relied be updated regularly, be updated regularly, on: Information required versioned, and curated for versioned, curated, and includes easy access.
|
From page 150... ...
, administrative records, or digital trace (Table 7-3) -- the following information should be saved, made public, or both.
|
From page 151... ...
Data collection questionnaire employed All details should be All details should be made (exact wording and skip curated for easy access. publicly available upon patterns)
|
From page 152... ...
Coverage error: Detailed technical reports Releasable technical undercoverage, overcoverage, should be prepared or reports should be duplications by key domains updated for each data prepared, updated, release, versioned, and for each data release, curated for easy access. versioned, curated and made permanently, publicly available on agency Website with DOI.
|
From page 153... ...
analysis of item nonresponse rate by question Percentage of failed edits: Details should be curated Releasable details should assessment of editing for easy access. be made publicly available procedures upon release of estimates, as part of technical reports.
|
From page 154... ...
Further, administrative records the code should be substitution commented to be readable by others and made available on request. Treatments for item The code for the The general description of nonresponse methodology used for the methodology used for treating item nonresponse treating item nonresponse should be retained and should be made available curated for easy access.
|
From page 155... ...
Variability of the official A technical report A high-level report estimates providing details of providing an outline of the estimation of the the estimation of the variability of the official variability of the official estimates, taking into estimates should be made consideration the effects of available to the public. nonresponse on the input datasets used, should be retained.
|
From page 156... ...
cAll code should default to being publicly available. Deviations (confidential parameters)
|
From page 157... ...
be made available to the to the program that would public. impact the continuity of data from one time period to the next DIGITAL TRACE DATA Data disposition: Descriptions of data Descriptions of data source of data elements and how elements and how description of data elements they compare from the they compare from the conceptual link between data information needed in information needed in and information needed for support of the official support of the official statistical product, including statistical product should statistical product should justification for use be retained.
|
From page 158... ...
impact the continuity of data from one time period to the next FOR BOTH ADMINISTRATIVE RECORDS AND DIGITAL TRACE DATA Transformations of variables The code(s) for the The description of the (e.g., creation of new variables various transformations various transformations for analysis through recoding used should be retained used and the reasons for or combining multiple items)
|
From page 159... ...
Methods used for variance Details and commented Details and commented estimation code should be curated for code should be made easy access. publicly available on request.
|
From page 160... ...
160 TRANSPARENCY IN STATISTICAL INFORMATION TABLE 7-3 Continued To be available only To be available externally, Information to retain or archive internally, to program staff to the public model form and related The form of the model, The form of the model, information the associated parameters the associated parameter and how they are estimates and how estimated, and assessments they are estimated, of the variability of the and assessments of parameter estimates the variability of the should be retained. parameter estimates should be made available to the public.
|
From page 161... ...
As a result, Table 7-4 focuses on record linkage or matching, a technique being employed at most, if not all, statistical agencies and which is one of the primary integration techniques currently in use. TABLE 7-4 Documenting Data Integration Issues To be available only To be available externally, Information to retain or archive internally, to program staff to the public Data files that were linked: A description, including A description of the identification of files the metadata, of the specific data files that description of files specific data files that were matched should were matched should be be provided routinely retained.
|
From page 162... ...
Documentation of Paradata While paradata are typically considered for survey data, we see no reason why paradata should not be available for administrative data as well, with analogous measures (see Table 7-5)
|
From page 163... ...
Response paradata reports: Technical reports should Technical reports should response latency be curated for easy access. be made publicly available key stroke studies on agency Website.
|
From page 164... ...
. TABLE 7-6 Archiving of Data To be available only To be available externally, Information to retain or archive internally, to program staff to the public Archiving of treated input data The input datasets used The input datasets and metadata (i.e., modified to produce the official used to produce the to account for failed edits, estimates (i.e., the official estimates (i.e., nonresponse, etc.)
|
From page 165... ...
The official estimates should be stored using persistent identifiers. Recommendation 7.1: The National Center for Science and Engineering Statistics and all agencies that produce federal statistics should, to the fullest extent feasible, document their data collection methods, their data treatments, their estimation methodologies, and assessments of the quality of their official estimates, and they should archive their in put datasets and their official estimates to support reproducibility and later reuse, as specified in the tables developed by the panel.
|
From page 166... ...
This will result in more sharing and reuse of input data and official statistical estimates and the methods used to produce them with the accompanying knowledge transfers within and among federal statistical agencies and with national statistical offices around the world. In this envisioned future, there will be greater interaction with the public, because today's user also wishes to make use of official statistics for nonstandard tabulations and as input to their own statistical models.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.