National Academies Press: OpenBook

Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies (2022)

Chapter: 4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users

« Previous: 3 Changes in Archiving Practices to Improve Transparency
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

4

Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users

INTRODUCTION

This chapter is concerned with three stages in the life cycle of official statistics. First, inputs are obtained, using a variety of methods. The inputs themselves, as well as the methods used to acquire them, need to be characterized, their quality assessed, and their fitness for use documented. The documentation must be made available to internal and external users, and it must be comprehensible. Second, these input data are processed, which can include data cleaning, transformation, coding, aggregation, analysis, model building, imputation, prediction, and so forth. Processing the data is primarily carried out through use of software code, but in some cases expert knowledge and manual processing are required. All processing steps, code, and instructions, including the assumptions made, must be documented. Finally, the data that are the output of the collection and transformation are published, along with pertinent metadata. The data must be usable to a wide spectrum of users, ranging from nontechnical users browsing an agency Website to sophisticated large-scale users wishing to use application programming interfaces to harvest, post-process, and display the data. Each of these segments of the statistical data life cycle is addressed in the three sections of this chapter.

Regarding the quality of inputs to official statistics, it is becoming more common for federal statistical agencies to use alternatives to survey data as inputs to statistics production. The primary alternatives are administrative data, from federal and state programs, and digital trace data from information stored on the Internet and through other technological means

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

of capturing information. This trend is a natural reaction to the increasing rates of nonresponse in sample surveys and hence to their declining quality, and to the increased availability of detailed, timely digital information about nearly every aspect of our lives. But it raises some difficult issues about how one assesses the quality of the resulting estimates, since there is no longer a framework such as total survey error (which one rarely has for all components for survey data either). The quality of the input affects the quality of the resulting official statistics.

In addition, if one is using a statistical model to combine estimates, which is typical for small-domain estimation (e.g., Fay-Herriot models), one needs to know the relative quality of the direct estimates and the model-based estimates one is combining in order to know how much weight each contribution should receive. However, the literature to date on how to assess the quality of administrative data, or digital trace data, is not fully developed, certainly not to the extent that it is for sample survey data. The associated transparency issue, then, is what should be retained when using administrative or digital trace data, to make whatever quality assessments one needs to make in order to permit use of such an approach to estimation. Further, if such quality assessments are carried out by the parent agency, these would need to be shared with the public as part of an effort to be transparent.

The second topic is the permanent retention of software code (and perhaps the software environment) used in data treatment or to implement the estimation methodology. While Chapter 3 discussed archiving of input datasets and official statistics, we have said little about retention of the software code for reuse, for investigation of the quality of previous estimates, or for checking computational reproducibility. It is worth emphasizing that retention of detailed computer code is essential for assessing (computational) reproducibility. There are a number of tools that have been developed that assist in such activities, including tools for collaborative software development, tools for retention of workflow history, and tools for providing a software environment in which to test code. These and other tools can be used for various purposes to support the retention and later reuse of the code originally used to produce a set of official statistics. While these tools are widely used in industry and academia, they are not yet widely used in the federal statistical system. Therefore, the panel thought that it would be helpful to describe what some of these tools can be used to accomplish.

The third topic addresses transparency in data dissemination and involves the extent to which the federal statistical agencies interact with users to find out what they would like to know about the production of a set of official statistics so that they may be best used. This includes what they would like to know about the data collection processes, the data treatment, the estimation processes, and the validation carried out on the official

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

statistics. Users may have strong opinions about the specific data products offered, including access to special tabulations, whether and where time series of the official statistics are available, and whether input datasets are available for study in federal statistical research data centers.

ASSESSING THE QUALITY OF INPUTS USED TO PRODUCE OFFICIAL STATISTICS

Assessing the quality of survey-based input data is a well-understood matter. The idea of total survey error,1 in place for many decades, lays out a framework for comparing different survey approaches to producing a set of official statistics by assessing the magnitude of the error coming from various sources. The various sources of survey error are initially divided into sampling error and nonsampling error, with sampling error summarized by the variance due to the sample design, and nonsampling error divided into coverage error, error due to nonresponse (both unit and item), measurement error, and processing error. These various sources of error contribute to the variance and bias of the resulting estimates.

The term “fitness for use” is often mentioned when making such comparisons. This is a somewhat more general notion, because it can include additional considerations, such as relevance and timeliness.2 However, even in this somewhat more general approach, estimating the biases and variances of the inputs going into the production of a set of official statistics is key.

In recent years, due to the higher costs of collecting survey data, primarily as a result of the increasing rates of unit nonresponse, other sources of data are increasingly being used in the production of official statistics.3 In particular, national statistical offices have increasingly used administrative data to produce official statistics. Administrative data are collected as a byproduct of the administration of a governmental program, often by collecting information to determine eligibility for the program, the size of the benefit, and information to help distribute the associated benefits. In addition to administrative data, also under consideration for use by statistical agencies are data collected from the Internet and other “technology” sources, including transaction data, social media entries, and sensor data, which is referred to here collectively as digital trace data. What is anticipated is that federal statistical agencies considering such data sources will first estimate the quality of the resulting official statistics using either survey data, nonsurvey data of the types described here, or (better yet) a

___________________

1 See Groves and Lyberg (2010) for an authoritative description.

2 See Brackstone (1999) for an excellent list of the factors that should be considered.

3 For an example, see Citro (2014).

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

combination of multiple sources exploiting the strengths of each individual source, with decisions of which approach to ultimately use based on cost-benefit considerations, taking all the factors of fitness for use into account. For these reasons—as indicated in Tables 7-2, 7-3, and 7-4—being transparent includes providing any information about the quality of the inputs used to produce a set of official statistics, since such information is at times not retained and would inform as to why the approach taken was decided upon, as well as the resulting impact on the quality of the official estimates.

For such an assessment to work, the agencies need to link the quality of the resulting official statistics using these various approaches to the information on the quality of the data from each individual source used. Consider the case of a statistical agency interested in changing from using a survey-based input dataset to one based on administrative data, for official statistics that are estimated means or totals. Such an agency should have to make the argument that some analog of total survey error for the proposed input data had either the same or lower levels of total error as the previous input data, or that the proposed method met other pressing needs (e.g., faster updates or much lower costs) despite an increase in total error. The need for transparency would dictate that all the information contributing to this comparison be made public.4

Because a statistical agency using administrative data for input into a set of official statistics will wish to assess the quality of the information collected by the administrative agency for this (often unintended) purpose, and because statistical agencies will not typically have access to any information about the errors that occurred in the collection of such data, it is important for the administrative agency to share any information they may have on the quality of their data. One concern is that the administrative agencies may not have the resources or the staff expertise to undertake an evaluation that provides the necessary information for this alternative use. Another concern is that the administrative agency needs to let the statistical agency know of any changes to their programs that could affect how they are used and their quality. To address this, interagency agreements may include additional data constructs (some type of paradata) that the administrative agency could record and share with the statistical agency.

As mentioned, in addition to administrative data, federal statistical agencies are also considering—and currently use to a modest extent—digital trace data in the production of official statistics. One example is that the Australian Bureau of Statistics is using supermarket scanner data (since 2011) and Web-scraped data (since 2016) in its estimation of that country’s

___________________

4 See Rancourt (2018), which explores what could be the framework for using administrative data more aggressively, and the embryo of a framework to measure the quality of the resulting estimates.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

consumer price index.5 As with administrative data, the use of digital trace data in support of a set of official statistics makes it important to evaluate their quality and the result of this on the quality of the resulting official statistics. This requires knowing details about the underlying data-generating process and the resulting fitness for use of the input data.

If official statistics are produced as a combination or integration of several sources of information, how these various sources are best combined will likely be a function of the quality of the various inputs.6 An example is the linear combination of direct (often survey) and indirect (model-based) estimates used in a small-area model, such as the empirical Bayes Small Area Income and Poverty Estimates models for states and counties used by the Census Bureau. In such a methodology with survey-based inputs, the proper weight given to the direct and indirect estimates depends on assessments of the relative sizes of their survey error.

In a sense, this kind of comparison is the analysis that Wolter and Hogan (1988) carried out to show that the error in unadjusted census counts was larger than the error in the adjusted counts (though in that case both approaches were survey based).7 The problem in doing this is that given the different origins of administrative data and digital trace data, it is not clear what might be meant by an analog to total survey error.8 In addition to the lack of clarity as to what the component parts are that need to be measured, or how to measure them, it may not be known whether or how such information should be combined.

How might one proceed? One can assess the quality of an input data set either externally, by comparing performance to that of some “gold standard,”9,10 or internally, by carefully assessing each step of the process through which the input data were collected. If there were a gold standard for the input data, one could sum up the differences and use that as a metric for quality assessment. Unfortunately, there rarely are gold standard comparison values. One is therefore limited to internal assessments,

___________________

5 For more detail, see https://www.abs.gov.au/articles/web-scraping-australian-cpi.

6 For further discussion of how various sources of information are combined, see Lohr and Raghunathan (2017), and Beaumont (2020).

7 See also Mulry and Spencer (1991).

8 For a more exhaustive review of data quality domains and dimensions, see Federal Committee on Statistical Methodology (2020).

9 Depending on what is viewed as a gold standard, one might have to account for differences in concepts, various kinds of mistakes, omissions, etc.

10 In fact, when we conduct a social inquiry (a survey) it is a controlled scientific experiment in which statisticians control the selection method so that variations and levels are caused by the socio-economic phenomenon. When we use administrative data, we lose this control and so the result is that we cannot ascertain with the same conviction that the changes and levels are due to the socio-economic signals; they could be due instead to the selection/participation process.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

accumulating what is known about the magnitudes of the various sources of error to which these estimates are subject. Further, it is not clear how these various magnitudes of error are to be combined into a single metric for comparing various types of estimates.

Administrative Data: Estimating Standard Error

It has now become standard practice for federal statistical agencies to use estimates of the standard error of survey-based official statistics, often in the form of coefficients of variation, for standard aggregates, for example, weighted means and sums. Given that considerable variability is often attributable to nonresponse (both unit and item), it is generally considered best practice to include in such estimated standard errors the contributions due to unit and item nonresponse. This is usually accomplished under the assumption that nonresponse is missing at random.11 The remaining sources of error identified in the total survey error approach are typically not estimated given the difficulty in measuring them.

How should one estimate the standard error of a weighted mean coming from administrative data? First, one can argue there is no sampling error, because one has a census of the population. However, the target population for a program can differ markedly from the population of those receiving the benefits from some governmental program, possibly because not all the eligible individuals are signed up, or some noneligible individuals have been mistakenly included in the administrative file, or some of those in the file may be duplicates, possibly as a result of living at multiple addresses. The result is likely some under- and some overcoverage. There also can be adjustments and imputations in the case of nonresponse. Further, there might be transformations needed to account for differences between what would be collected in a survey and what is collected through administrative data—such as whether there is agreement for the time window of interest and who or what precisely are the members of the target population—and it may not be obvious how to make the necessary transformations from the administrative world to the survey world. For these reasons, it is not a safe assumption that administrative data from a closely related federal program provide information representing the populations of interest to support production of a set of official statistics (see Cunnyngham, 2020).

And how should one estimate the standard error of a weighted mean drawn from digital trace data? It is typically the case with such data sources that some population members have no chance of being represented while others are represented multiple times, with no clear way to determine how many are omitted or overcounted. There are also questions regarding

___________________

11 For a review of methods to measure total survey error, see Biemer (2010).

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

what the intended measurements on the target population are and how the digital trace data correspond to these intents. If a statistical agency was responsible for the data, and the response/participation mechanism was not understood, there would likely be nonresponse studies to find out if the nonrespondents have similar characteristics to the respondents. However, this does not often happen for administrative or digital trace data.

As is obvious, when the official statistics are model-based estimates formed using survey data—say, some type of regression estimates—the errors in the dependent variable and in the predictors can affect the errors in the estimated regression coefficients and therefore the fitted values. In the case of small-area estimation, the standard errors of the domain estimates are estimable if the standard errors in the inputs are (see Prasad and Rao, 1990). However, it is now becoming common, as in the case of the Small Area Income and Poverty Estimates program, for some of the predictors used by federal statistical agencies in their models to come from administrative data. In such cases, given the lack of information on the quality of these data—or, for that matter, the quality of digital trace data or other sources of nonsurvey data—estimating the quality of official estimates can be challenging. This is another reason why transparency is important. These are issues and complexities that have not been fully resolved and will require further research.

In the case of administrative data, some of the recent contributions to this research area are detailed by Zhang (2012), Reid and colleagues (2017), and Oberski and colleagues (2017). Zhang (2012) notes “a clear lack of statistical theories for assessing the uncertainty of register-based statistics” and then provides a good first step for doing so. He divides contributions to error into two parts. The first part is the contribution to error from the use of an individual input dataset in producing a set of official statistics. Errors here may be due to measurement deficiencies, or they may be due to the degree to which the input data are or are not representative of data from the target population. The second part, which is divided into these same two pieces, contributes to error in the integration of multiple sources of data and so is concerned with issues such as using post-stratification weights and matching. Reid and colleagues (2017) expand Zhang’s approach by including a third part, the uncertainty from estimation. They point out

Designing statistical outputs that use administrative data creates many new challenges because we have to give up direct control over many processes, including population definitions, collection methods, classifications, and data editing. Each administrative source has its own particular problems that must be understood both for our own design work and to assure the final users of the data that our outputs are fit for purpose. When we use

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

administrative data instead of a traditional survey, we need new processes, such as data integration, and recoding or adjusting administrative variables, which can introduce new types of errors. (p. 478)

At the present time, there is no uniform agreement on what to provide to summarize the quality of the information from an administrative data source. Amaya, Biemer, and Kinyon (2020) provide more operational guidance in recommending the use of what is referred to as the total error approach. Even if the approach taken is not comprehensive, whatever information contributes to an understanding of total error is valuable. We have mentioned some of the issues above. It is anticipated that in many cases there will only be minimal information on, say, how often respondents made use of some type of assistance in filling out an eligibility form, which does not get one very far but is a start. Even if this information does not directly lend itself to some assessment of total error, having any information available on data quality might contribute to better use of administrative data when used in combination with other data. One interesting question is whether some sort of summary error assessment could be used in estimating the best linear combination for a small-area estimation model where the direct estimate is from administrative data (see also Benzeval et al., 2020).

Web-scraped Data: Quality and Other Issues

If the primary dataset used to develop a set of official statistics is scraped or otherwise collected from the Internet, the analogy regarding total survey error in association with survey data is much less clear even than that for administrative data. The population of transactions on the Internet or the population of those making transactions is difficult to understand because (1) it is sometimes difficult to know how many times a single individual is represented in such a database, and (2) it is likely impossible to know how many people are not represented. (On the other hand, there may be no nonresponse or misresponse.) We do believe that such data will be found to be extremely useful as predictors in statistical models used to produce official statistics, but the assessment of the quality of such official estimates may take longer to understand. The current state of research is summarized in Amaya, Biemer, and Kinyon (2020).

Statistics Canada has a set of principles for determining when and how to use Web scraping, which includes a gold standard that could be developed to measure quality aspects of a Web-scraped data source.12 The National Center for Education Statistics (NCES) has a good set of frame information for all public13 and most private14 elementary and secondary schools and

___________________

12 See https://www.statcan.gc.ca/eng/ourdata/where/web-scraping.

13 Common Core of Data: https://nces.ed.gov/ccd.

14 Private School Survey: https://nces.edu.gov/surveys/pss/.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

postsecondary institutions15 in the country, which facilitates measuring coverage errors in a Web-scraping project about schools and universities. These sources include data that could be compared with the same identifiers on Websites to check agreement rates or correlations with high-quality data. Inferences can then be drawn on the quality of data unique to the Web-crawling collection.

One hurdle that arises in assessing the quality of much digital trace data is the lack of good population frames like those noted from NCES above. If one is estimating regression coefficients for a model-based estimate that are assumed to be constant throughout the population, and one is using digital trace data for one or more predictors, so long as the dependent variable being fit is from high-quality frame data one can assess and understand the error of such an estimate. On the other hand, if one is trying to estimate a population parameter using only digital trace data, for example using some type of ratio estimate, the error properties of the estimate may be more difficult to assess. It seems difficult to avoid having the data contaminated by undercoverage and overcoverage. As another example, some researchers have tried to make use of search engine data as predictors of dependent variables of interest (e.g., Google flu trends). But various dynamics in how often the public makes use of specific search terms can have a negative impact on the predictive strength of such predictors, which often cannot be anticipated. These issues remain topics of ongoing research.

The Federal Committee on Statistical Methodology (FCSM) has been involved in a multiyear project, at the request of the Interagency Council on Statistical Policy (ICSP), to develop an evaluation framework for integrated data that will help to identify documentation needs. Several workshops have taken place and reports issued (see FCSM, 2020). The committee’s final report, A Framework for Data Quality (FCSM-20-04), provides important information on defining the components of quality for datasets, documenting components of data quality, and identifying threats to data from these various sources.

Czajka and Stange (2018) provide a review of international standards and guidelines in the reporting of quality for integrated data. In their report, they review the literature on reporting standards in support of transparency for administrative data. They are supportive of the work of Zhang (2012), and they review what is referred to here as digital trace data, where very little work has been done. They also cite the United Nations Economic Commission for Europe (UNECE) Quality Task Team (UNECE, 2014) as the most significant work on the topic. Internet transaction data and social media data represent a substantial amount of data of generally unknown quality or provenance. Czajka and Stange (2018) argue that while there are many kinds of data to be mined from the Internet, the reporting framework is not well developed, and they suggest that much more research and discussion is needed. Until common agreement is

___________________

15 Integrated Postsecondary Education Data System: https://nces.edu.gov/ipeds.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

reached on a reporting framework that helps data users determine a dataset’s fitness for use, data producers should provide a discussion of the known statistical strengths and weaknesses for each statistical application, opportunities for undercoverage, degree of nonresponse, and opportunities for misresponse.

TRANSPARENCY IN PROCESSING, SOFTWARE DEVELOPMENT

As a matter of principle, all data collection, data processing, and data transformation must be made as transparent and open as possible. Currently, most data collection, processing, and transformation is conducted using software, although some manually constructed survey instruments, manual entry of paper forms, and expert input to data cleaning may also be used. This section primarily discusses software tools that assist in making the software code that conducts these actions transparent, but will also elaborate on how to make manual steps as transparent as possible.

Transparency of Computer Code

A survey instrument may be programmed in Blaise,16 hosted on open platforms such as LimeSurvey17 or on commercial platforms such as Qualtrics,18 or developed in some other custom software designed specifically for the purpose of a particular survey. Data cleaning and transformation may be conducted in SAS,19 SQL, Python, and other tools, and model analysis and imputation may be implemented in the above-named software as well as in Stata, R, SPSS, or other programming tools. For any particular portion of the collection and processing, code may range from several hundred lines to hundreds of thousands of lines of code.

What all of these scenarios have in common is that at least some of the code is developed in house or by contractors, specifically for the purpose of the statistical agency. Even though SAS and Qualtrics are commercial software platforms, the particular code being run or the particular specification of a survey instrument is clear and remains under the control of the agency. At any point when data are collected, transformed, or used in a statistical model using such code or specification (which will be referred to as “code” from now on), a snapshot of the code should be preserved. One reason for the importance of this is that since methodology programs are modified relatively often, knowing which version gave rise to a set of estimates is not always clear, and being transparent about the specific version

___________________

16https://blaise.com/blaise/about-blaise.

17https://www.limesurvey.org/.

18https://www.qualtrics.com/.

19https://www.sas.com/en_us/home.html.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

used is crucial in assessing reproducibility. The evolution of the code may also be of interest, primarily for auditing purposes, allowing internal and possibly external users to identify when a change was introduced that might have affected data quality or fitness for use.

One of the most fundamental tools is a version control system. Briefly, a version control system allows anyone to modify and save any part of an arbitrarily large software system (or its documentation), without affecting the work of others. Such capability is typically used to create a process (and tools) to enable other experts to review and test the change and, potentially, leads to that change being incorporated into the released version of the system delivered to end users.

This practice and associated tools have been refined over time to make the entire review, integration (among the many changes made by all developers working on the project), and quality control (in the form of regression testing) into a sophisticated release management process. Such release management increases transparency, as all decisions—starting from a developer’s change to feedback by the reviewers, and the tests that have been run by the quality team—are all fully documented by the associated tools. Further, in case any issues arise, this kind of release management makes it easy to track how and why every change was made. Needless to say, without the tools designed for release management, this would be an impossible task. More specifically, continuous integration has been a critical innovation allowing a developer to immediately check if his or her changes might cause malfunctions if integrated into the system. Such checking is performed by regression testing, where automated tests are run on the changed system to ensure that the outputs produced are still correct.

While transparency was not the main objective for commercial software development, it was essential for open-source development. In fact, most of the ideas related to the concept of open science appear to have originated in open-source development. Unlike in older commercial software projects, where development teams used to be co-located, open-source developers are often distributed all over the world. So, there is an inherent need to come up with suitable governance strategies and collaboration tools to produce software in this environment. The key principle in such distributed collaboration is to explicitly document and make public all individual and group decisions and be able to incorporate input from outsiders. The mailing lists for developers typically describe all new requirements for software, as well as changes in governance or in release processes. More technical input is collected through issue trackers that allow anyone to report a problem, a suggestion, or code improvement.

The concepts and tools related to a version control system—including code review, continuous integration and testing, and collaboration and issue tracking—are not magic solutions but require discipline and good

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

practices to be used effectively. Importantly, many organizations build upon these fundamental building blocks to devise good practices for each task. For example, release management typically involves a variety of quality gates (e.g., limiting the number of outstanding issues), criteria for feature selection, and defined roles and responsibilities of the different parties in commercial and open-source projects alike.

Modern version control software, and more generally software configuration management software, has increased substantially in usability and availability. While stalwarts such as Concurrent Versions System were developed in the 1980s, more modern systems emerged in the early 2000s (Subversion, now Apache Subversion, was at one point the most popular system). As of 2021, “Git,” created in 2005 by Linux creator Linus Torvalds, appears to be the most popular open-source version control software. Many commercial versions also exist, such as Mercurial (created in 2005), ClearCase (1990), and Visual SourceSafe (1995).

The use of such systems within statistical agencies and their contractors, which appears to have increased in the past 10 years, is not documented in a consistent way, as far as the committee could ascertain. As noted by Rob Sienkiewcz (Census Bureau) in a presentation, the Census Bureau’s Longitudinal Employer-Household Dynamics (LEHD) has been consistently using (internally available) version control since 2001, and Subversion since 2003, and it uses a formal release policy for all code used to generate published statistics, though such code is not publicly posted. It claims to be able to identify all the code used in the production of the Quarterly Workforce Indicators for each release going back 15 years. The Decennial Census’s Disclosure Avoidance System has been using Git internally for all of its development, using software by Github and Gitlab, and has published the code as used for its 2010 demonstration products in full.20

As the LEHD case demonstrates, it is possible to use such systems for audit purposes, and as the Decennial Census’s Disclosure Avoidance System shows, it can promote transparency and public inspection of novel technologies. The LEHD code also successfully avoids use of any hard-coded but secret parameters, a feature that is incorporated into each code review of released code, making it technically possible to easily publish the code.

Given the focus on data for federal statistical agencies, we focus on tools that have been developed in the area where software and data intersect. For example, contemporary machine learning and statistical techniques require a plethora of software packages that are to be used in a single data workflow involving data collection, cleaning, and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. One example of the use of supervised machine learning at

___________________

20https://github.com/uscensusbureau.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

the Bureau of Labor Statistics is to map the words people use to state what their occupation is, or what injuries they have experienced on the job, into standardized categories.

This multitude of tools may be complicated to install and set up on a workstation. Many such applications are, therefore, shipped as preinstalled virtual machines. This saves time and frustration for someone trying to build upon or reuse existing approaches. Furthermore, with data analysis it is important to tie code and corresponding data output or graphics in a single document. An open-source tool mentioned above, Jupyter Notebook, is a Web application that allows users to create and share documents that contain live code, equations, visualizations, and narrative text and support data workflows integrating tasks ranging from data collection to the presentation of results. Such tools are typically used in conjunction with virtual machines and version control systems. This allows reproducibility even of the most complicated data workflows.

It appears that few of the federal statistics organizations use such tools and practices to engage outsiders or even to coordinate the work within or among the agencies. Using these tools would increase the transparency of the decision making in these agencies prior to the release of data products. Notably, software-as-a-service providers, such as Github and Atlassian, provide an online delivery and sophisticated integration of many of the tools noted above as well as training needed for their effective use. This significantly lowers the entry barriers for federal statistics agencies to transition to a more reproducible and transparent infrastructure.21

The Decennial Census program staff uses internal and external versioning systems (Github and Gitlab) for documentation of the code used to perform the computations needed in carrying out the census. Further, they make code publicly available as it is used, up to what can be feasibly released, given various confidential and privacy parameters. Many other statistical agencies use Github for similar purposes.

To evaluate the role these tools might play in the future in the federal statistical agencies, there will be a need for greater access to computer science expertise and an examination of best practices in programming and data curation. The current versioning system of choice is Git, which was created in 2005, so these tools are not recent developments; their value to industry and academia is clear. The additional computer science expertise needed can be acquired through new hires or through various consulting arrangements, with different advantages stemming from each approach.

___________________

21 The following Web pages contain useful information outlined in this section: https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control; https://en.wikipedia.org/wiki/Issue_tracking_system; https://www.lucidchart.com/blog/release-management-process; https://www.docker.com/; https://github.com; https://bitbucket.org; https://jupyter.org.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

The support of the ICSP and more informal interagency cooperation will be valuable, as some agencies (likely the larger ones) get started earlier on investigations into the role such tools can play and the benefits that doing so provides.

Recommendation 4.1: Agencies that produce federal statistics, including the National Center for Science and Engineering Statistics, should review and make a priority of adopting modern information technology tools that assist in collaborative software development and documentation of workflow and methodology.

This is important, because transparency through computational processes is as important as the transparency of other processes, and also because it will make their transparency efforts more efficient and facilitate internal reproducibility and evaluation.

However, it will not be sufficient to simply collect incomprehensible (if functional) code. Code can contain copious amounts of documentation, can be structured to be more easily understandable without documentation in English grammar (using programming style guides), and can be accompanied by high-level and detailed documentation and software-agnostic specifications. All such documents and practices increase both internal and, when published, external transparency. Many agencies spend many person-hours crafting the latter, but little is known about the use of programming style guides.22

As the Decennial Census’s Disclosure Avoidance System has shown, it is possible to develop new systems, including those pertaining to sensitive disclosure avoidance systems, in a public and transparent manner. Doing so supports and encourages transparency, while allowing for computational reproducibility in some cases.

At the other end of the data life cycle, agencies generally publish survey instruments (questionnaires), and these have been used successfully to allow interpretation of historical data collections. Applying the principles of transparency laid out here suggests it is not only the hard-coded questionnaires that are of use, but also the coding instructions for such questionnaires. It is in this context that standardization of coding (e.g., exporting questionnaire specifications in Data Documentation Initiative format) becomes a requirement for greater reproducibility. While it is unlikely that a single researcher will re-implement an entire federal survey, one frequently sees researchers re-using certain questions (e.g., the Current Population Survey [CPS] demographic module) or re-using entire supplements (e.g., the Contingent Worker Survey supplements to

___________________

22 For an example of coding style guides, see https://google.github.io/styleguide/.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

the CPS as replicated by the RAND-Princeton Contingent Work Survey; see Katz and Krueger, 2019). While the availability of re-usable questionnaire specifications is not a necessary requirement to be able to replicate a federal survey, it can greatly reduce the development costs.

Logging

In addition to retaining the versions of the code that were used to collect, treat, and transform the data, an actual “transcript” or log of the actions conducted by the code may also be relevant. From the academic literature, it is well known that all features of the computational environment can affect computations.23 It is thus critical to record not just the code used to collect or process data, but also the environment in which such code was executed, and (subject to some reasonable constraints) to record logs of such execution as well. In some environments, keeping such logs may be legally required to document authorized access. While it is possible to keep extremely detailed computational traces (e.g., capturing low-level hardware modifications is technically possible), it is generally considered sufficient to keep logs of the software as it is executed.

In certain secure environments within statistical agencies, all use of the software generates a logfile for audit purposes, but in the context of transparency and reproducibility, simpler logfiles may be sufficient. Compared to the data being generated, logfiles are generally much sparser and thus easier to archive together with the generated data, acting as a form of metadata, or in some cases, paradata.

Special purpose statistical software (SAS, SPSS, Stata) has such ability built in, though it is not always enabled. General purpose software such as Python or C++ will not, in general, generate logs unless explicitly programmed to do so. All use of software as it is being processed should generate logs, and this should be enforced through coding style guides. Whenever code is run to produce published output, all logs in the processing sequence should also be archived.

Modern literate programming tools (Knuth, 1992) often promise greater transparency by having the (English grammar) text interspersed with computer code. Examples include Jupyter Notebooks, Rmarkdown documents, and Sweave documents (all using R). This is not unique to such newer tools. Coding style guides at LEHD from 2005 required that programmers use processes similar to Sweave to intersperse legible English

___________________

23 For an example of how this can affect computations within a given programming language, see for instance Gould (2011). For the importance of accounting for these issues for privacy, see Garfinkle and Leclerc (2020). For an older but still relevant description of the issue, see McCullough and Vinod (1999).

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

language and mathematical formulas with functional SAS code.24 Some of the newer systems (such as Jupyter Notebooks) may be subject to unreliable “out of order” processing, leading to irreproducible results—the exact opposite of the intended outcome (Wang et al., 2020).

Transparency of Manual Processes

It was mentioned earlier that not all data collection or transformation is conducted purely by code. Paper forms, on which data are manually entered, may still be collected in certain circumstances. Qualitative responses, such as industry or occupation descriptions, may be manually coded into industry and occupation codes. In specific cases, expert input may be used to correct or fill in unlikely or implausible data.

These scenarios have two features in common. First, the data can be compared before and after the manual intervention. Thus, documentation can at a minimum point to preserved and possibly archived data that can be inspected to see the effect of manual interventions. Second, none of the manual processing is done in a void. Employees are trained, using instructions, manuals, and guidance. These constitute “human programs”—instructions that humans read, interpret, and implement. And any such instructions are almost always stored as electronic documents, which in turn can be versioned, archived, and made public using similar tools as outlined earlier in this chapter.

Recommendation 4-2: To facilitate transparency, agencies that produce federal statistics are encouraged to develop coding style guides, and to make available documentation and specifications for software systems, subject to any security concerns. Where possible, code (for example used for data collection or processing) should be made publicly available, subject to redaction or removal of confidential parameters, and logs of processing sequences should be archived. Manual processing steps should be clearly identified and documented, and any instructions or guidance given to the staff conducting such manual processing should be archived and made as transparently available as possible.

This might entail a limited amount of redaction, but the existence of any redaction for privacy-preserving purposes should be considered an integral part of the documentation.

___________________

24 Also see Lenth and Hjøsgaard (2007), and for the later StatRep package, see Arnold and Kuhfeld (2015).

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

Of Special Relevance to Chapter 5

A statistical activity is what statistical agencies do to design, collect, transform, estimate, integrate, disseminate, or archive data under a statistical program (e.g., a statistical survey or census). An example might be top-coding the values for some variable collected by a survey. Given this, a statistical process is how each of these is achieved within each of the subject-matter areas in each of the federal statistical agencies. Statistical software languages (e.g., SAS, SPSS, Stata, or R) might be used to carry out the edits. The source code for such an editing procedure would contain detailed documentation for the top-coding edit procedure.

These activities and processes are part of what constitutes statistical methodology. The methodological design corresponds to an activity, instructing as to what is to be accomplished by some part of a statistical program. These instructions are sometimes known as specifications. How that activity is carried out, in theory, is an algorithm, the logical steps needed to achieve the requirements in the design. And these logical steps are also sometimes known as procedures. The source code, for example, is the process in practice when it is automated. The source code includes the steps required by the algorithm and the constraints of the programming language being used. Source code written in any of dozens of programming languages can each perform the same algorithm, and many algorithms can satisfy one design. For example, the problem of sorting a set of values alphabetically or numerically—a design—can be accomplished by several well-known algorithms. Each one can be implemented using one of many programming languages.

A statistical program is an implementation of a set of methodologies put into practice (from designs to algorithms to source code), and when one documents those methodologies one is documenting the statistical program. But, as will be discussed in Chapter 5, documentation and metadata are the same thing, so the source code for the automated portion of a statistical program is part of the (detailed) metadata for that program. Exactly where the source code fits into the overall metadata framework for a statistical agency depends on several factors, and these are also discussed in Chapter 5. One important point to make here is that metadata specifications should be part of the software design and development process.

FACILITATING USER INTERACTION WITH STATISTICAL AGENCIES

Transparency in data and methods is important to many users and their needs, especially to know whether the official estimates are fit for a particular use. Similarly, the cost-benefit assessments discussed in Chapter 2

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

are not possible unless agencies know which measurements are important to the public and the impact various errors would have on various applications of the official statistics. Consequently, part of any effort to provide greater transparency should be taking a more dedicated and systematic approach to understanding what users need.

More broadly, in order to understand and meet user needs in data products, documentation, dissemination systems, and archiving, agencies must develop mechanisms to solicit more frequent input from their user community and facilitate ongoing dialogue with them. A number of the federal statistical agencies have given limited effort to understanding what their users need in terms of transparency, accessibility, and usability of data products to enable optimal use of official estimates and associated input datasets. But there have not been many surveys or other systematic efforts to collect information about users’ needs or assessments of satisfaction with documentation, Web pages, and dissemination platforms.

While there is a considerable amount of valuable information on the Web pages of the federal statistical agencies, these resources are often not easily found. This is an important consideration if one is to ensure that metadata and underlying computer codes are going to become useful to users. Moreover, input from data users has not usually been sought before the introduction or redesign of new products, dissemination systems, and Websites. As a result, data users have found that some of the interfaces developed to facilitate access and use of estimates are not intuitive or user friendly (an example is the second version of the U.S. Census Bureau’s American FactFinder) and Web pages are not easy to navigate. For a more relevant example for the National Center for Science and Engineering Statistics (NCSES), the two expert NCSES users mentioned in Chapter 2 said that new users often need substantial mentoring from more seasoned users to be able to find and access the data they need to answer research questions of interest. Given that funds and staff time are limited, agencies can more efficiently and effectively target these resources if user input is obtained both prior to and during the development of new products, documentation, dissemination systems, and Website redesigns.

One of the primary challenges agencies face in addressing these gaps in user interaction is first identifying the members of their user communities. Agencies can overcome this obstacle by establishing ongoing data user groups through a variety of mechanisms. For example, agencies could offer data users the opportunity to sign up to receive announcements or a periodic newsletter through email, or they could establish online communities—similar to the Census Bureau’s American Community Survey Online Community—that users could opt to join for free. The advantage of an ongoing data user group is the ability to communicate directly with users at specific times and with specific information or requests. In contrast,

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

notices that are simply posted on an agency’s Website will reach some members of their user community, but only if they happen to visit the Website during a particular timeframe.

Once an agency has established an ongoing data user group with a contact mechanism (email, online community, etc.), there are four activities it could undertake to increase and enhance interaction with its data users that would benefit the agency:

  1. Statistical agencies could periodically survey their user communities to determine the kinds of problems they are experiencing accessing and using agency statistics, including ways that each agency could make its Web pages easier to navigate, or ways they could provide the estimates or microdata to better facilitate various often-used analyses, including time-series and cross-sectional analyses.
  2. Statistical agencies could survey their user communities to solicit specific input before changes are made to data collection techniques, estimates, data products, dissemination systems, or Web pages to ensure that data users’ needs will still be met after proposed changes are implemented; and they could also involve members of the user community in reviewing and providing feedback as these changes are actually implemented.
  3. Statistical agencies could create a mechanism, such as an online community or forum, that enables members of their data user group to communicate directly with each other—posing and answering questions, providing solutions to user-identified problems, and raising issues for community member feedback. Such a mechanism could reduce the user-support burden on agency staff and be particularly useful for isolated analysts who need more guidance in finding, accessing, and using particular agency data series.
  4. Statistical agencies could meet regularly with representatives from their user communities to engage in more in-depth dialogue about ongoing issues and potential future improvements to estimates, data products, documentation, dissemination systems, and the structure and navigation of agency Web pages. This is similar to the various federal advisory committees that several of the statistical agencies currently meet with. Is it possible that they could be made more robust to address user needs? Could they meet more frequently? Could they be enlarged?

Other possible activities that should be considered include:

  1. systematically analyzing comments and questions posted to identify common themes and needs for improvement;
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
  1. conducting a “use case” analysis of different types of users and evaluating the utility of the Website for each type;
  2. giving users a place to share not only advice and comments but code and user-generated datasets; and
  3. instituting outreach to encourage new use of the data.

In addition, there might be value in reaching out to data journalism groups like Investigative Reporters and Editors/National Institute of Computer-Assisted Reporting, which hosts active listservs where they share tips for navigating the confusing data interfaces from federal agencies.25

Restructuring Web pages can be a complicated task for the agencies. There are a variety of types of users that one wishes to accommodate, and different structures might be preferable to different user types. For example, a long-time NCSES data user might wish to understand what changes have been made to the most recent version of a survey; a journalist might wish to simply download the current value to support her article; a non-NCSES data user might happen onto NCSES data found in data.gov and as a result visit NCSES’s Website to search across topics and subtopics to learn more about an issue of interest; or an analyst might want easy access to an internal archived dataset to reproduce a specific statistic to check on computational reproducibility. One must also be cognizant that there will always be users who are comfortable with the current Website and navigation, and they will find that their experience is disrupted with the implementation of a new structure. However this is addressed, accessibility is crucial. Is information that is provided findable and usable by users? Whatever system is employed to give users access, it must be navigable in an intuitive way.26 A topic related to the above is the availability of various technical and methodology reports, metadata constructs, and other ways of providing the public with access to other information on the data treatments and methodologies used in the production of a set of official statistics. This can include codebooks and research reports that discuss attempts to make improvements to different aspects of the computations carried out. In NCSES, such reports are often available internally in draft form, but either they are never reviewed and therefore are not made publicly available, or they are viewed as being too technical for public release. We understand that the demand for such documents may be limited, but for a small subset of the user community such documents can be extremely important for providing detailed information to researchers on how data treatments were implemented and how estimates were produced, and in

___________________

25https://www.ire.org/resources/listservs/.

26 We have provided Recommendation 6.7 to NCSES concerning their need to better understand their users’ data needs and preferences; it can be found at the end of Chapter 6.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

informing researchers about which aspects are currently being examined for improvements. Further, having access to these will allow other users to gain an appreciation for the care that underlies these programs and their statistical products.

Finally, it is obvious from the content of this chapter that many innovations are coming on line now. For example, the Coleridge Initiative has sponsored a Kaggle challenge on rich text analysis that employs algorithms that will substantially improve the ability to see who in the research community has used various datasets and what they have published. This will create a real-time opportunity for agencies to see how their data are being used, which in turn will help them become more responsive. This helps all statistical agencies meet a requirement in the Evidence Act to get feedback from the public on the utility of their data. More such innovations are on the horizon, and they will have an impact on transparency.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×

This page intentionally left blank.

Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 73
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 74
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 75
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 76
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 77
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 78
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 79
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 80
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 81
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 82
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 83
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 84
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 85
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 86
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 87
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 88
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 89
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 90
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 91
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 92
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 93
Suggested Citation:"4 Assessments of Quality, Methods for Retaining and Reusing Code, and Facilitating Interaction with Users." National Academies of Sciences, Engineering, and Medicine. 2022. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies. Washington, DC: The National Academies Press. doi: 10.17226/26360.
×
Page 94
Next: 5 Metadata and Standards »
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies Get This Book
×
Buy Paperback | $35.00 Buy Ebook | $28.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Widely available, trustworthy government statistics are essential for policy makers and program administrators at all levels of government, for private sector decision makers, for researchers, and for the media and the public. In the United States, principal statistical agencies as well as units and programs in many other agencies produce various key statistics in areas ranging from the science and engineering enterprise to education and economic welfare. Official statistics are often the result of complex data collection, processing, and estimation methods. These methods can be challenging for agencies to document and for users to understand.

At the request of the National Center for Science and Engineering Statistics (NCSES), this report studies issues of documentation and archiving of NCSES statistical data products in order to enable NCSES to enhance the transparency and reproducibility of the agency's statistics and facilitate improvement of the statistical program workflow processes of the agency and its contractors. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies also explores how NCSES could work with other federal statistical agencies to facilitate the adoption of currently available documentation and archiving standards and tools.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!