6

Making the Practices of the National Center for Science and Engineering Statistics More Transparent

DESCRIPTION OF NCSES PROGRAMS

The National Center for Science and Engineering Statistics (NCSES), one of the 13 principal federal statistical agencies, is located within the Social, Behavioral, and Economic Sciences Directorate within the National Science Foundation. NCSES’s goals are the collection, interpretation, analysis, and dissemination of objective data on the science and engineering enterprise. The center is responsible for collecting data and publishing official statistics on (1) research and development (R&D); (2) the science and engineering workforce; (3) U.S. competitiveness in science, engineering, technology, and research and development; and (4) the condition and progress of science, technology, engineering, and mathematics education in the United States. NCSES fulfills its mandate primarily through the collection, analysis, and dissemination of statistical data in support of research on these topics.

NCSES’s survey portfolio includes 15 ongoing surveys and one upcoming survey (see Table 6-1). Of these ongoing surveys, 10 are annual, 4 are biennial, and the periodicity for one is to be determined. Ten of the ongoing surveys are performed under competitive commercial contract, and the remaining five are performed under negotiated interagency agreements with the Census Bureau.

In addition to data tables and other data products (see below), the primary survey outputs include several types of analytical reports. These include

TABLE 6-1 NCSES’ Survey Portfolio

NOTE: The National Training, Education, and Workforce Survey (NTEWS) is a new NCSES survey currently under development with plans for the survey to be performed under a negotiated Interagency Agreement with the Census Bureau.

• InfoBriefs, which highlight findings from recently completed surveys¹ or reports on narrowly defined topics;
• Congressionally mandated reports, such as Women, Minorities and Persons with Disabilities in Science and Engineering and Science and Engineering Indicators. For the latter report, NCSES produces the congressionally mandated Indicators Summary Report and cycle-specific thematic reports;
• Special recurring reports, such as Doctorate Recipients, based on the Survey of Earned Doctorates (SED) from U.S. universities, and National Patterns of R&D Resources: Data Update, which is released annually;
• Working papers;
• One-pagers, e.g., InfoChart;
• Infographics, e.g., InfoBytes; and
• Methodology reports (currently only available by request).

Due to resource limitations and time constraints, the methodology reports are given a lower priority for internal review and dissemination.

___________________

¹InfoBriefs also provide special analysis across survey datasets or on a special topic. See the following examples: https://www/nsf.gov/statistics/2019/nsf19306/, https://www/nsf.gov/statistics/2019/nsf19300.

The data products include data tables, which are preformatted and generated by an underlying database, released with technical survey information, public-use microdata files, state profiles and academic institution profiles, interactive data tables, and restricted-use files.

More recently, NCSES has added an interactive data tool, which has a landing page where users can select a topic or type a search term. The user can also go directly to some surveys from the Build Table option. A unique feature that has been built into the search capability is one based on synonyms: instead of being limited to exact character matches, related terms can be identified using this capability, and when a user sees something of interest, he or she can select a detailed panel of relevant information. In the near future, the interactive data tool will publish the metadata application programming interface that drives this site, adding a translation layer that publishes the metadata to the Data Documentation Initiative (DDI) standards, with automated updates to Data.gov. Also, in the near future, this data tool will support an interface for downloading public-use microdata from NCSES’s integrated data system.

There is also Data Explorer,² recently released, which facilitates exploring metadata at various levels, including measures, definitions, and variables. Finally, there is an external tool called SESTAT that provides access to public data from the Survey of Doctorate Recipients (SDR) and the National Survey of College Graduates (NSCG).

NCSES has record-level data for the following programs: Early Career Doctorates Survey (ECDS); federally funded research and development centers (FFRDCs) R&D and facilities; Higher Education Research and Development (HERD) Survey; NSCG; SDR; SED; Survey of Federal Funds for Research and Development (Fed Funds); Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions (Fed Support); Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS); and Survey of State Government Research and Development.

Restricted survey data are available for ECDS, NSCG, SDR, and SED in the NCSES data enclave, managed by NORC at the University of Chicago on behalf of NCSES, and referred to as the Secure Data Access Facility. Researchers may apply to NCSES for access to data available within this facility. Restricted data from the Annual Business Survey (ABS) and Business Enterprise Research and Development Survey (BERD) can be accessed by researchers who apply to the Census Bureau for microdata access through federal statistical research data centers (FSRDCs). Restricted-use data for NSCG, SDR, and SED are also available through the FSRDCs.

NCSES employees who are involved in the production of surveys conducted by the Census Bureau using Title 13 or Title 26 data survey frames

___________________

² For details, see https://ncsesdata.nsf.gov/explorer.

(BERD, ABS, NSCG, SDR, and SED) can access the microdata for those studies through the Survey Sponsored Data Center (SSDC). The SSDC is a physical space within NCSES that is treated as an official Census Bureau location; the space is maintained in accordance with Census Bureau policies concerning physical and cyber-security protections for sensitive data. NCSES employees who work in the SSDC must have Special Sworn Status with the Census Bureau. Any output that is removed from the SSDC, even to be shared with other NCSES employees, must go through Census disclosure avoidance review to ensure that no information protected under Title 13 or Title 26 protected data is disclosed. The SSDCs are not designed to support research activities by NCSES employees, even those that use these same data. Census’s expectation is that research projects will be undertaken in an FSRDC. With the recent expansion of virtual access for both the SSDCs and the FSRDCs, this program may be more amenable to providing the interagency transparency that is necessary to support improvements in quality and reproducibility for the NCSES data programs.

More broadly, transparency is likely to be complicated for any agencies that contract out surveys to others, including other federal agencies with their own policies, if the funding agency does not receive either survey responses or low-level aggregates needed to implement their own reproducibility and transparency-related procedures. Contracts and interagency agreements need to provide for transparency. The increasing use of remote access should make that more achievable in the future.

As mentioned above, NCSES produces public-use data files, with microdata for HERD, NSCG, SDR, GSS, FFRDC R&D, and facilities. Public-use files are typically delivered from the survey contractor and reviewed and released on the NCSES Website. Additionally, the NSCG and SDR public-use data files are reproduced in house by the data team as a validation effort to check the contractor-delivered files. Several survey data files are only available under restricted access, to protect respondent confidentiality. For these data, NCSES may grant secure access to restricted-use microdata files through the Secure Data Access Facility.

NCSES also has acquired management oversight of the single-application portal known as ResearchDataGov.³ The Census Bureau initiated the single-application portal effort and transitioned the leadership to NCSES in 2020. This portal achieves a key goal of the Foundations for Evidence-Based Policymaking Act of 2018, which is to develop a single, user-friendly online application and process for requesting data assets found in restricted data for developing evidence.

Publication Standards Utilized By NCSES

NCSES currently operates in accordance with its publication standards, as represented in their internal (not publicly available) document, “Statistical Standards for NCSES Publications.” Below is the current list of standards regarding documentation of data treatments, methods, and dissemination. The four primary standards are these:

1. Include general statements in the data sources and limitations section discussing the following:
   • Sampling error (if applicable).
   • Non-Sampling error
     • In general (e.g., nonresponse, errors in processing, false information provided by respondents, errors in the questionnaire, coverage or frame errors, measurement errors)
     • Any nonsampling error suspected to be associated with a particular survey.
2. Errors that could influence the results of the analysis must be explicitly addressed (e.g., high imputation rates or editing rates for a variable used in the analysis).
3. A description of any statistics that differs from our standard measures (means, medians, totals, ranking, percentages, ratios, percentiles [25th, 75th, etc.]) must be included within the text or as a footnote.
4. Any methodology that differs from our standard methodology with respect to weighting procedures, or handling of non-response must be investigated and justified. A description of the methodology must be included within the text or as a footnote.

When compared with the panel’s best practices tables in Chapter 7, which detail what a comprehensive approach to transparency would entail, this document leaves some things unstated regarding the archiving of record level data, details of the data collection process, where and for how long

___________________

³https://www.icpsr.umich.edu/web/pages/appfed/index.html.

the official statistics are archived, for how long input data are retained, and what metadata are used to describe these retained files. As a result, there is much left to the discretion of the survey manager, data manager, and others at NCSES involved in the collection, processing, storage, and dissemination of data.

TRANSPARENCY FOR EXTERNAL USERS OF NCSES SURVEY OUTPUT

To assess NCSES’s transparency, the panel was interested in the extent to which NCSES provided information on its own Website, over and above its publication standards, to inform NCSES’s external user community about various details concerning its statistical programs. Such details could include information about survey designs, the survey instruments used to collect responses, details about how to instruct the field interviewers, the extent of nonresponse and failed edits, how the survey weights were computed, the estimation methodology used, and the variability of those estimates. For this purpose, the panel decided to look at information concerning the BERD, SED, HERD, and ECDS surveys. These four surveys were chosen because of their prominence, because they represent surveys of both institutions and people, and because they are conducted by private-sector contractors as well as by the Census Bureau.

Business Enterprise Research and Development Survey (BERD)

The NCSES Web page for BERD describes the BERD survey design, which includes both simple random sampling and probability proportional to size sampling within strata, where the measure of size is based on historical research and development funding estimates. The stratification is based on research and development activity (with the most important strata being for businesses with research and development greater than $3 million), payroll, and the North American Industrial Classification System code. The sampling frame is the Business Register, maintained by the Census Bureau. BERD uses multimode data collection, a Web instrument used by 99 percent of respondents, and a paper booklet returned by mail by the remainder of the respondents.

Regarding data treatments, all data submitted are subject to several hundred automated edit checks. Approximately two-thirds of these edit checks are designed to find arithmetic errors and logically inconsistent responses, such as failed balance checks. There are also edits for outlying responses. Most failed edits are addressed using imputation; some are addressed by contacting the company. On the Website, there is also an overview of survey quality measures that is provided and there is a summary of

recent changes to the survey. The degree of and treatment for unit and item nonresponse are provided, though details are left to tables in the annual reports. Coverage error is described as minimal. There is measurement error due to various differing interpretations as to what research and development activity is and due to differing accounting practices. There are also links to the survey instruments used and to the historical series of official estimates going back to 1953.

Estimation methodology is explained in the technical notes of annual detailed statistical tables and detailed in the annual methodology reports. For that reason, these BERD Web pages themselves do not detail the estimation methodology, where the weights used are a function of the sampling weights and degree of nonresponse. It would be helpful to inform users as to how the weights are calculated, but such details are also left to annual reports. Finally, there is a discussion of the comparability of the survey estimates over time given various changes to the data collection methodology, with the only substantial issues arising in 2007 and 2008.

Survey of Earned Doctorates (SED)

The NCSES Web page on SED describes this census of earned doctorate recipients. It also presents a discussion of the automated editing that is used. The only imputation for missing data that is conducted is for the month used to calculate the age of a doctorate and the time taken to complete a degree. There is also information on quality measures, which includes information on the extent of undercoverage and the degree of unit and item nonresponse, and there is some information available on the degree of measurement error. The questionnaire is also available for users to examine.

Understandably, there is no direct access to individual data due to disclosure laws, but there are two interfaces that users can use to build tables. First, for SED data from 1958 onward there is an interactive data tool that can create custom tables of the number of doctorate recipients by demographics, discipline, and institutional characteristics. Researchers can also access SED microdata through the Secure Data Access Facility and the FSRDC. The Restricted Data Analysis System (RDAS) tool, which protects confidentiality, provides access to SED data not otherwise available using the interactive data tool, which is restricted to cross-tabulations of the above-mentioned variables. However, there are plans to also access SED cross-tabulations using the RDAS tool in the near future.

Higher Education Research and Development (HERD)

The NCSES Web page on HERD contains key information about this census of educational institutions, including the questionnaire for multiple

years, descriptive and methodological information such as types of edit failures and treatment for nonresponse, and quality metrics. The frame consists of all nonprofit postsecondary institutions in the United States, including Guam, Puerto Rico, and the U.S. Virgin Islands, that granted a bachelor’s degree or higher in any field and which spent more than $150,000 for research and development in fiscal year 2018. Respondents may choose to respond by filling out a paper survey or by using the Web-based data collection system; 99 percent of respondents use the Web-based system. Respondents could be recontacted to address failed edits based on prior reporting patterns and unexplained missing data. Imputations for missing data are based on the previous year’s data and on the data from peer institutions.

Coverage error is described as being minimal. The extent of item and unit nonresponse is also provided, and historical data are available dating back to 1972. There are some modest comparability issues over time due to changes in the census, and analysts are encouraged to contact NCSES to find out whether these issues would affect an intended analysis.

Early Career Doctorates Survey (ECDS)

The ECDS is a relatively new survey that gathers in-depth information about individuals who earned their first doctoral degree (Ph.D., MD, or equivalent) in the past 10 years and work in U.S. academic institutions or FFRDCs. The survey, described on the ECDS Web page, uses a two-stage design, the first stage being for institutions and FFRDCs, and the second covering doctorate recipients within institutions or FFRDCs. A primary stratification splits the institutions and FFRDCs into separate strata.

The selection of academic institutions is then stratified using a Carnegie classification and depending on whether the institution contains a medical school. The sampling of institutions within these strata uses “probability proportional to size” sampling based on the number of degrees historically granted per year, the number of doctorate researchers, and the number of faculty. The institutions selected then provide lists of their doctorate recipients, which are sometimes anonymized to protect privacy. Once these lists are made available, proportional allocation is used to identify the number of sampled respondents. Web survey and computer-assisted telephone interviews are offered to these respondents, the Web survey being selected more than 99 percent of the time. The questionnaire is available on the NCSES ECDS Web page. Data treatments include editing and logical and hot-deck imputation for item nonresponse. The estimates are weighted totals that account for differential sampling rates and nonresponse. Post-stratification is employed so that marginal totals agree with control totals.

For external users of the survey results, on the Web page reasons are given for possible issues leading to under- or overcoverage, and the rates of

unit nonresponse are provided. Users are informed that for those interested in analyzing microdata, access to restricted data can be arranged through a licensing agreement. They are informed of the survey data quality by the release of the sampling variances and are provided with the rates of nonresponse as well. While nothing quantitative is available concerning measurement error, some cognitive testing is mentioned that was carried out to minimize this source of error.

EASE-OF-USE OF INFORMATION FOR ANALYSIS PURPOSES

The panel was also interested in ease of access to and use of the information provided, especially the ease of analysis of the official estimates themselves. Therefore, the panel asked two expert users of NCSES data for their views on the uses of the official estimates provided and associated information on issues such as the quality of the data. One expert was Anne Marie Knott, the Robert and Barbara Frick Professor in Business at the Olin School of Business, Washington University. In her presentation, Knott stated that NCSES provides important data for researchers like her who are interested in the support available for research and development. She is particularly interested in data from the Business R&D and Innovation Survey (BRDIS) (the predecessor of BERD). She initially accessed BRDIS data from a Census Bureau research data center.

One problem Knott noted when using these data in a research data center was the lack of documentation. She said it was difficult for her to determine what the precise questions were that were responded to as well as what the responses available for each question were. The documentation that was available added up to roughly 200 pages, so it was hard to find what one needed. Also, she noted, there were many versions of some variables.

In addition, Knott noted that when she was working from the publicly available estimates from the NCSES Web page, like many users of such data she was interested in examining trends over time. To carry out the desired time-series analysis, she had to merge various tables over time and then link the merged file with other covariates over time to try to relate various trends to certain explanatory variables. But such analyses were not easy, because the table numbers for different cross-tabulations change over time, and she sometimes found it difficult to determine which tables to merge. A third issue, Knott said, was that there were jumps in the data that one would assume corresponded to quantities that should be smooth over time. These jumps may have been due to changes in the survey, but documentation of such changes was difficult to find. As a result, it was hard to assess the quality of the estimates.

The second expert the panel consulted was Kimberlee Eberle-Sudre, director of Policy Research for the Association of American Universities,

which comprises 63 leading research universities in the United States that collectively receive 61 percent of all federal expenditures. In her role, she accesses NCSES data at least weekly to track research funding received for use in the association’s federal advocacy work, and she also uses NCSES data in the association’s membership processes. Eberle-Sudre finds the data from NCSES to be essential for her work. The data have contributed substantially to the advancement of their advocacy for research at universities and colleges, she noted. As a frequent user, especially of HERD, she has become familiar with many of the nuances of both the Website and the data. She stressed that it does take some time to become comfortable with different aspects of the data tables. She believes that if NCSES modified the table presentation, it would facilitate use of the tables, especially by newer users.

Eberle-Sudre added that the cross-tabulations from the tool that replaced WebCaspar are not as user friendly as they might be. She cited three aspects of this problem: (1) one cannot save selected institutions, and as a result, each time someone wants to form a table of data on institutions, each institution must be manually selected, which can be tedious; (2) the order matters in terms of which cross-tabulation variable is selected first, second, etc.—for example, if a user chooses institution names first and then another variable, the next option may not be available since expenditure type was not selected earlier; and (3) it is difficult to locate variable descriptions, changes to variables, institutional names, and types of funding. Eberle-Sudre sees a need for a section on each program’s Website that clearly lists all important changes from one survey implementation to the next and possibly another section that lists all changes to the issue under study.

While the data tables are very useful, Eberle-Sudre said, another problem is that they are designed specifically for gaining a broad descriptive view of higher education research. When a user needs to view institutional-level data or data on specific funding by federal agency, the tables are not particularly useful. She sees a need for more user involvement in the development of appropriate data tables and user interfaces.

Given the difficulties that Eberle-Sudre and other newer users have experienced, before major changes to the presentation of data are undertaken it might be useful to form a user community or network of users, which new users could turn to for mentorship and guidance. This would be particularly useful for isolated analysts of NCSES data series. (More detail is provided in Recommendations 6.1 and 6.6 below.)

PRIORITIES FOR NCSES

This panel study can be viewed as having two tasks. The first task is to examine the information that NCSES provides about its programs both internally and externally, especially the input data and official statistics and

the methods used to produce those official statistics, to assess the degree to which they are currently transparent and their methods are reproducible. Further, if such an assessment demonstrates that there are some areas in which NCSES could be more open, the panel’s task is to help NCSES determine what initial steps could be taken, in the most efficient manner possible, to bridge the gap between what is currently made available and what could be made available.

Second, as expressed in our statement of task, the panel has been asked to address many of the same questions for the entire federal statistical system that it has been asked to address for NCSES. As a result, in Chapter 7, as well as in previous chapters, we present a number of recommendations for ways the federal statistical agencies can more comprehensively document methods and archive their official statistics and input data. Because it is a federal statistical agency, those recommendations also apply to NCSES. However, many of those recommendations will require resources and considerable time to implement. NCSES requested that this panel also provide recommendations that are mainly relevant to NCSES that would help it begin to become more transparent in the next few years, without the addition of substantial new funding. Therefore, in the following we present recommendations designed to help bring about greater transparency for NCSES in the short term.

To begin, the panel was impressed with a number of innovative steps that NCSES has already taken to permit users to access the data underlying its cross-tabulations from its surveys to form user-requested tabulations. Such personalized use of survey data could be further enhanced by learning more about how these tools can be further refined to make them even more helpful to the public.

Some of the methodology reports from NCSES have not been made publicly available on their Website due to time and budgetary pressures. Examples are reports on data collection, data treatment, estimation procedures, and assessments of the quality of input data and official estimates. In order to better communicate with the public, NCSES needs to keep the public abreast of methodological improvements, investing more in the production and availability of such reports on the NCSES Web page.

We understand that NCSES does have a policy and process for retention of its official estimates, but we are not aware of any policy that dictates where and for how long the associated input datasets are retained. While the following recommendation pertains to NCSES, all federal statistical agencies should have a records retention policy in place, approved and monitored by the Office of Management and Budget.

As is true for most of the federal statistical agencies, NCSES has made only modest use of shared metadata standards, in particular in its data documentation and in its exchange of official statistics and the associated methods. Given the benefits gained from sharing its official estimates with its international partners, it seems particularly beneficial for NCSES to make greater use of a metadata standard like SDMX to facilitate the transfer of its official estimates to other international statistical offices.

Following on our discussion in Chapter 4 of the importance of retaining paradata, NCSES should examine whether and how its programs use paradata, and based on that should examine when and for what time period paradata should be retained.

Given the discussion in Chapter 5 of the need to establish more regular contact with its user community, NCSES should take steps to institute greater interactions with that community to improve its perspective on how use of its data products can be facilitated.

• Organize regular meetings with broad user community representation; and
• Through surveys and direct interactions with users, identify ways to improve the transparency, accessibility, and usability of NCSES estimates, data products, documentation, and dissemination systems, including the structure and navigation of the agency’s Website.