3

Benefits and Costs of Transparency: Views from Three Statistical Agencies

Susan Offutt (member, steering committee) began the session by discussing broad issues of the costs and benefits of transparency, noting that transparency and reproducibility are important to the federal statistical system. One benefit from being transparent to external users and researchers is that those users may suggest improvements to federal statistical products. There are also internal benefits to agencies of being transparent, including the facilitation of technology transfer. However, she noted, increasing the degree of transparency is not costless, since the costs include the time necessary for documentation and archiving. Therefore, agencies might consider both costs and benefits in thinking about how much transparency to strive for.

Offutt added that this consideration is not just for survey data. While this session will focus on survey data and the associated statistical products, the use of other types of data sources is increasing. She noted, for example, that the Panel on Methods for Integrating Multiple Data Sources to Improve Crop Estimates, of which she is a member, is studying forecasts of crop yields at the U.S. Department of Agriculture by looking not only at survey data but also remote sensing data that look at crop progress. To combine the data from these different sources, statistical models are used. The use of models gets complicated, which increases the costs of transparency, she said, but perhaps there are gains from having an informal external review that is possible with transparency. Thus, one needs to understand the tension between what agencies can afford to spend on transparency and the demands of their users. In addition, it is important to understand whether there are technological solutions that can make transparency more cost efficient.

Offutt introduced the session’s three speakers, all of whom are high-level government officials with key insights into this tradeoff: Jennifer Madans (National Center for Health Statistics [NCHS]), Sally Thompson (Bureau of Economic Analysis [BEA]), and John Eltinge (U.S. Census Bureau).

JENNIFER MADANS: BENEFITS AND COSTS OF TRANSPARENCY

Jennifer Madans began by stating that at NCHS there is a long tradition of maximizing transparency in all of its activities. She said she does not have a strong interest in a formal cost-benefit analysis of the value of transparency because there are very clear benefits to the agency. However, there are costs that grow as the data and methods in question become more specific and detailed, so proactive decisions could be made about how much information should be made publicly available. Even with the efforts NCHS makes to maximize transparency, every now and then there is a user who thinks the agency has not gone far enough.

Overall, Madans said, NCHS views transparency as one of the necessary costs of being a federal statistical agency. The commitment to transparency sets federal statistical agencies apart from other data collectors. She acknowledged that the costs to the agency are substantial, and that as more money is spent on transparency, there is less money that can be spent on other activities, such as the scope and amount of data that can be collected, the scope and number of products that can be released, and the speed of both collection and release.

Aside from the responsibility of openness as part of the federal statistical system, Madans said that a major benefit of transparency to NCHS as an agency is to support agency credibility. This, in turn, maximizes the utility of the information because it is then trusted. The approach also protects agency independence because when people do question a result, the first thing they do is criticize the methods. NCHS opens the books and explains clearly what was done.

NCHS has always had a focus on releasing as much data as possible for use by the public, both through public-use files and with restricted access through research data centers. The agency’s publication series is meant to support the data releases and to highlight key findings. NCHS produces few projections and results of sophisticated models; these have not been a focus of the agency. Rather, NCHS’s focus has been on the collection and dissemination of information as quickly as possible to as many users as possible.

Madans described the major NCHS data collection systems: the National Health Interview Survey (NHIS), the National Health and Nutrition Examination Survey (NHANES), the National Survey of Family Growth, the National Vital Statistics System, and the National Health Care Surveys. She noted that NHANES presents some unique confidentiality issues because

a health examination is administered as part of the survey. Clearly, NCHS collects large amounts of survey data of national interest for which transparency is important. However, she pointed out that NCHS does not collect the data itself: it contracts with either the U.S. Census Bureau or various private companies. As a result, there are some small “black boxes” in the operation where the contractors know precisely how something is carried out but NCHS does not know. NCHS is trying to reduce or eliminate those holes, she said. However, in general, the agency has a good grasp of its methods and the methods of its contractors.

Madans noted that the National Vital Statistics System operates under somewhat different circumstances. NCHS does not have a lot of control over the collection of those data at the source because vital registration is a state function. For this system, NCHS has what is now a 100-year collaboration with the 57 jurisdictions that collect vital records. NCHS supports standardization of what is collected and how it is collected across the jurisdictions, and documents the processes used.

Lastly, Madans described NCHS’s set of surveys called the National Health Care Surveys. Respondents include physicians, hospitals, emergency departments, outpatient departments, and ambulatory surgery centers, who provide records of their activities. The data from the records are not what traditionally are thought of as administrative records, because NCHS abstracts from the records and so there is some modification of the original responses. NCHS is trying to move to electronic health records, but it will have even less control of what is entered into those records. She noted that when people look at their own medical records, they are amazed at what is in there and what is not. It is hard for NCHS to provide documentation for aspects of these surveys because NCHS does not collect the original information.

Madans turned back to the broad issue of transparency. NCHS attempts to standardize and comprehensively document data collection processes, but that can be challenging. Data collection and documentation generally go through the following stages: (1) planning—telling people what NCHS is going to collect; (2) development—developing and documenting the actual survey instrument, how it is administered, what the interviewer guidelines are, etc.; (3) field work—collecting the information and documenting what happens in the field; (4) data preparation, editing, and imputation, including edit programs that may have been written a long time ago that are rarely examined; (5) file release with documentation; and (6) published reports and papers published by others. For the last point, the key question is whether one can follow the references and get back to the documentation that is on NCHS’s Website. Transparency can be considered separately for each stage. In her opinion, NCHS gets very good grades for transparency in planning and this has further improved recently. It also gets good marks in trans-

parency in development and in published reports, and excellent marks in transparency in field work and in file release. However, it could do better in transparency in data preparation, editing, and imputation.

Madans said her main worry is about issues of data quality of which the agency could be unaware. There are only certain things for which NCHS will do an in-depth evaluation, and sometimes those are done after the data have been collected and so it is not always clear how to document that stage of the process. An example is the collection of data regarding vitamin D in the NHANES survey. The surveys were showing a population-wide decline in vitamin D, but the change appeared to be due to the tests used. To address this, NCHS did a crossover study, which was feasible because the respondents’ blood samples were banked, allowing the agency to redo all of the tests using different techniques. Madans posed the question of where this kind of situation should be documented. In this case, NCHS wrote a report, which is on the agency’s Web page, but is that sufficient? Also, for survey data, NCHS does not know what may be a hidden example of mis-response. More broadly, where should NCHS put its resources? For example, should NCHS go back to the 1985 NHIS to document the metadata when very few people in the last 10 years have needed that information? It is possible, she said, that funds should be moved from documentation to evaluation.

Madans added that she is aware that this workshop is concerned about the extent to which federal statistics are reproducible, and she raised several questions. First, is transparency a goal only in that it supports reproducibility—that is, is the goal only that someone can do what was done using the published data and methods? What is the appropriate starting point for such a discussion? For NCHS, the documentation of public-use files and analyses are designed so that analyses based on the public-use files can be reproduced. Is this sufficient? And is that really of interest to anyone other than to show that the number that was published is correct? It does not say anything about the way that the number was calculated or why the question on the survey was asked the way it was. She raised several additional questions: What should be documented about how a public-use file was created? Does an agency go back and let people recreate a public-use file? What can be useful about analysis-specific decisions?

A key issue, Madans said, is how an agency deals with missing data. NCHS has tried to document that process, but sometimes it is not straightforward. How far back should one go in the file to try to document how particular decisions were made? Should this apply only to key indicators or to the content of all NCHS publications? What about reports and papers not published by NCHS? Is it sufficient for the agency to maintain the information and make it available on request or does the information have to be available on its Website? Does an agency have to make everything known

about a data element available every time it is reported, or is it sufficient for such information to be available on the agency Website?

One can spend a lot of time worrying about reproducibility, Madans said, but she questioned whether that is really the primary benefit of efforts toward transparency. She repeated her idea that NCHS is being transparent to add to the credibility of the agency. Although complete transparency should result in reproducibility, she asked whether transparency in decisions about what data to collect and how to collect the data should be required even if those decisions are not directly related to reproducibility. She raised another series of questions: How much transparency is enough and in what areas? Is transparency in all aspects of the data collection process important for replication? What are acceptable ways to document data collection processes? Must everything be publicly available even if the user community is small for some data elements? Is it sufficient to maintain information and make it available on request? Madans noted that people often take the NHANES methodology or the NHIS questionnaires and try to use them in another setting. In such cases, information on how the tools were implemented to get comparable results can be useful. This situation raises the question of what an agency might be able to do for such users in order for one to determine whether the results are consistent.

Madans said that she has no answers to these more general questions about tradeoffs. NCHS will maintain a commitment to transparency so that others can reproduce what it does, but mostly so that others understand what the agency is doing and so that it can maintain credibility in the user community.

In closing, Madans acknowledged that there are costs to NCHS’s approach and the agency has accepted those costs. The costs associated with transparency affect the scope of NCHS’s data collection, data releases, and publication programs. The agency would like to minimize those costs, she said, and it would appreciate guidance from the statistical and wider community on how to better define what those boundaries are, what necessitates transparency, and the extent of transparency.

SALLY THOMPSON: BENEFITS AND COSTS OF TRANSPARENCY

Thompson began her presentation by noting that the various agencies in the federal statistical system all operate differently. She describes some of what is done at BEA, acknowledging that some of what she is saying will not generalize. One major difference is that BEA does not directly collect data as much as some of the other agencies; it spends more time and energy combining information from various sources. She said that BEA gets a lot of its data from the U.S. Census Bureau and much of its price data from the Bureau of Labor Statistics, but it also gets data from the U.S. Departments

of Agriculture, Defense, and the Treasury; the Federal Reserve Board; the Internal Revenue Service; and the Energy Information Administration.

Source data are very important to BEA, she said, and they must be timely and of high quality, which are often in conflict. She quoted from the agency’s mission statement:¹ “The Bureau of Economic Analysis (BEA) promotes a better understanding of the U.S. economy by providing the most timely, relevant, and accurate economic accounts data in an objective and cost-effective manner.”

Thompson listed some of BEA’s best-known data products: the national accounts, which have components of gross domestic product (GDP), personal income, and corporate profits; international accounts, including balance-of-payment accounts, trade in goods and services, and foreign direct investment; industry accounts, including input-output accounts, GDP by industry, and travel and tourism accounts; and regional accounts, including GDP accounts by state and personal income information by state and local areas. She said that BEA data products help with economic analyses of businesses and households and support decisions about monetary policy, fiscal policy, and state and local planning and allocation of funds. Thompson continued that there are various mechanisms for ensuring objective statistics. They include statutory requirements and various directives and standards. At the bureau level, there are security and release procedures, the transparency of methods and sources, and regular review of revisions.

Regarding statutory requirements, the Information Quality Act of 2001 directed OMB to issue government-wide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by Federal agencies.”² Thompson reminded workshop participants that the resulting OMB guidelines directed federal agencies to establish and issue their own guidelines toward this end. BEA’s Information Quality Guidelines were implemented in 2002; they are available on the BEA Website and are updated regularly.³

Thompson noted that in terms of directives and standards, there are OMB Statistical Policy Directives, which identify minimum requirements for federal principal statistical agencies when they engage in statistical activities. There are also measurement standards and various classification standards. She noted that OMB also makes an effort to conform its directives to various international standards.

___________________

¹ See https://www.bea.gov/about [November 2018].

² See https://obamawhitehouse.archives.gov/omb/fedreg_final_information_quality_guidelines [January 2018].

³ See https://www.bea.gov/about/policies-and-information/information-quality [January 2018].

For security and release, Thompson detailed BEA’s six procedures: (1) limited access to sensitive estimates to those with a need to know; (2) physical and computer security necessary to limit access to those with a need to know; (3) ensuring no BEA employee uses (or gives the appearance of using) prerelease information for personal gain or inadvertently provides prerelease information to the media or other unauthorized individuals; (4) regular “best practices” training for staff handling and processing sensitive data; (5) “lockups” for principal economic indicators; and (6) releasing data according to an announced schedule, which minimizes the risk of prerelease access, provides a clear separation of statistical agency and policy officials’ statements, and ensures simultaneous release to all users and the public.

Thompson then turned to the transparency of sources and methods. For source data, the Source Data Improvement and Evaluation Program was established in the 1980s. Since BEA relies on others for data collection, this program monitors the needs of all of BEA’s programs for data provided by federal agencies, and it documents the data sources underlying BEA’s estimates. For concepts, methods, and estimation procedures, BEA strives to make information readily available. For each of the 2,500 separate accounts it maintains, there may be a separate type of model-based estimation, imputation method, extrapolation method, or seasonal adjustment method. She stressed that BEA does a lot of analysis with the input data, and transparency facilitates an understanding of the concepts and methods. In addition, she said, for each estimate there is usually a set of external gross domestic product (GDP) forecasters who try to forecast GDP; to do so, they have to have studied all of the methodology documents and know where to get the same source data for input into their estimation methodology. Thompson said that she does not know of any other agency that has such an “industry” that works on its data. When the data and methods are not transparent, this user group lets them know. When BEA publishes a number that is different from ones suggested by the forecasts, the agency gets phone calls and e-mails. She noted, however, that although BEA documents what data it is using and the estimation methodology used for its various statistics, the agency does not publish detailed formulas relating source data to final estimates.

She noted that BEA publishes regular revision studies in its Survey of Current Business journal. These studies examine the accuracy of the estimates over time and test for any systematic overstatement or understatement.

Thompson then discussed ensuring the integrity of BEA products in three ways. The first is by addressing measurement challenges. This involves improving the accuracy of more timely GDP estimates. BEA collaborates with source-data agencies to accelerate and improve the data used to estimate GDP. For instance, BEA collaborated with the U.S. Census Bureau to

achieve major reductions in revisions to GDP estimates because the U.S. Census Bureau was able to accelerate the publication of trade data, inventory data, and quarterly services data. BEA targets the timeliness, coverage, and quality of macroeconomic indicators leading to improvements in timeliness and accuracy of measures of economic growth.

There is also evidence of residual seasonality in some of BEA’s GDP estimates, she explained, for which BEA has a three-stage plan to resolve. First, the agency is revising existing BEA source data and aggregations for origins of residual seasonality. Second, it is modifying the GDP estimation process based on the results of the first-stage review. And third, it is developing and releasing current-quarter GDP estimates that are not seasonally adjusted. The results of this plan will be reflected in the statistics released as part of the comprehensive update of the National Income and Products Accounts in July 2018.

BEA is also developing new products as a way of ensuring integrity, Thompson said. Some of this work includes the creation and development of various satellite accounts, including research and development, which have now been incorporated into BEA’s core accounts; health care spending; arts and culture; travel and tourism; small businesses; and outdoor recreation.

Thompson explained that another way BEA is working to ensure integrity is by exploring new data sources. The current push to exploit “big data” presents challenges in terms of using them for official economic statistics. Examples include data from Zillow and from the Athena Health System. Also, BEA is exploring the use of more credit card data. Thompson ended her presentation by noting that it is unclear what the relevant transparency issues are for these data, and she offered several questions: How representative are the data? Do the concepts match those necessary to measure output, prices, employment, etc.? Do the data provide consistent time series and classifications? Is it possible to bridge gaps in coverage? How timely are the data? How cost-effective are the data? And what confidentiality issues arise and how limiting are they?

JOHN ELTINGE: COSTS AND BENEFITS OF TRANSPARENCY

John Eltinge opened his presentation with the proposition that there is a relationship among the goals of transparency, reproducibility, and replicability with quality/risk/cost profiles and stakeholder value in the production of statistics. In particular, he said, this relationship is going to be critical when one considers the benefits and costs of transparency standards in terms of the alignment of transparency activities with various elements of those underlying profiles.

Eltinge said his presentation will focus on three elements to make this connection: a qualitative description of transparency, reproducibility, and

replicability; some elements of a conceptual framework; and some specific types of benefits, risks, and costs, extending some of the ideas put forward by the two previous speakers.

In general, Eltinge said, one can think about transparency for federal statistics in terms of three questions: Is enough information being provided to users so that they can gauge the quality, risk, cost, and value profiles of the products that an agency produces? That is, can those users use the information provided to assess the implications of those features for their particular use? Can one use assessments from users to give the U.S. Census Bureau some insights into ways in which it can improve its products and expand its product lines?

For reproducibility, Eltinge said there is one critical question: Could an independent analyst who looks at the description of what has been done for the original production, along with the input data (though that is not clearly defined), reproduce what was published? In contrast, for replicability, there is a different question: Could one get basically the same inferences based on independent measures from new independent data collected from the same population? The issue of replicability for the U.S. Census Bureau appears in a number of cases in which there may be “house effects” when one is looking at a collection and processing activity that the agency previously contracted out to one organization that was later contracted out to another organization. The U.S. Census Bureau often sees a difference in the products, and it is often attributed to such house effects. He noted that there may be some very interesting Bayesian interpretations for each of these components of transparency, reproducibility, and replicability.

In general, Eltinge said that for all U.S. statistical agencies, their mission is to produce high-quality information on a sustainable and cost-effective basis. To do so, agencies could align the operational definitions of transparency, reproducibility, and replicability with the mission statement. In addition, he said, agencies would be transparent in communicating with their stakeholders about the costs and risks of transparency, reproducibility, and replicability. Agencies might consider users’ expectations of being transparent on risk factors and cost structures. This question is especially important when an agency encounters financial challenges and has to make decisions as to whether to drop a product series, reduce its sample size, or take other steps that may end up reducing quality.

In addition, Eltinge said, when one thinks about quality, the customary criteria, according to Brackstone (1987),⁴ are accuracy, relevance, timeliness, comparability, coherence, and accessibility. He noted that accuracy for

___________________

⁴ Brackstone, G.J. (1987). Statistical issues of administrative data: Issues and challenges. In Proceedings of the Statistical Uses of Administrative Data: An International Symposium organized by Statistics Canada, November 23–25.

survey estimates is addressed by the assessment of total survey error, which gets into population coverage, sampling error, incomplete data, specification error, and measurement error. There are also modeling errors, adjustment errors, the use of disclosure limitation methods, etc.

Eltinge added that the context of estimates is also important. Sources of variability would be considered as to whether they are controlled in the design or included in the model. Also, he said, there is the distinction between reproducibility and replicability and which sources of variability one is conditioning on. There are also political implications as to what factors are shown to influence which outcomes.

Eltinge then discussed the prospective benefits, costs, and risks of transparency, reproducibility, and replicability. The benefits include quality improvements, which derive from a better understanding of the data through stakeholders’ use, in that transparency is a precondition for data improvement. This factor is particularly important when integrating data from multiple sources and managing a complex supply chain of statistical information. Risks can be reduced by reducing the likelihood of tunnel vision and reducing the frequency of undetected blunders. He also noted that cost increases as a result of quality improvement efforts will result from the development and implementation of more efficient designs and models.

Turning to risks, Eltinge noted that there will be contract management and various institutional dynamics. In addition, there are the usual risk factors that accompany the development of standards and requirements. These risks result from formal compliance being divorced from substance and regulatory capture and accountability issues. Also, he said, it may be the case that the underlying science and practice are not yet sufficiently mature to support a refined standard. The risks specific to transparency are calcification, Eltinge said, because the additional cost of transparency increases barriers to innovation, and the perceived loss of intellectual property rights can discourage cutting-edge investments. He said there is also the chance for unreasonable criticism. To mitigate these risks, one can be aware of these possibilities and identify their impact on stakeholder use and value.

Eltinge then noted some specific costs of transparency. First, there are direct labor costs, which involve documentation and curation of code, along with refined datasets. There are also opportunity costs for one’s most productive employees, he said, and there is a cognitive and operational burden for users, especially if transparent information is not well calibrated with the stakeholder information base.

Eltinge offered four suggestions for agencies with respect to increasing transparency and reproducibility for statistical agencies. First, anchor transparency in an agency’s mission statement. Second, have the goal of practical improvement in long-term stakeholder value. Third, align efforts with the dominant sources of variability. And fourth, work out transpar-

ency and reproducibility improvements. He added that it is especially important in terms of big data to consider, in addition to documentation, conducting sensitivity analyses to indicate how much various assumptions matter.

This page intentionally left blank.