Measuring Research and Development Expenditures in the U.S. Economy (2005) / Chapter Skim
Currently Skimming:
8 Survey Management and Adminstrative Issues
8 Survey Management and Adminstrative Issues
Pages 152-176
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 152... ...
That model envisions a federal statistical agency with a small staff of federal employees consisting primarily of subject-matter experts, supported by a limited number of specialists in survey management, survey design, the cognitive sciences, information technologies, and data analysis. In this model, survey operations are outsourced to other organizations that have a comparative advantage due to their size and concentration.
|
From page 153... ...
Within these departments, the independence of the statistical agency derives from its distinction from the parts of the department that carry out enforcement or policy-making activities. The SRS division is fairly deeply embedded inside NSF, given that the agency has a fairly flat structure for a government agency (see Figure 8-1)
|
From page 154... ...
The mission of SRS extends beyond the mission responsibility of social, behavioral, and economic sciences. The division informs the operations of the Of rice of the Director and the Directorate for Social, Behavioral, and Economic Sciences.
|
From page 155... ...
However, since SRS contracts out all of its survey work, it is left with a very thin staff. There is little or no opportunity for growing a bench of expertise in the necessary methodological specialties, including mathematical statistics, cognitive sciences, and survey design, such as exists in the larger statistical agencies.
|
From page 156... ...
. That report recommended three courses of action to alleviate the staffing and skill shortages: augment staff expertise through professional development actlVItles.
|
From page 157... ...
to prepare and maintain survey methodology and technical documentation and provide this information in a comprehensive methodology report and (2) to recommend quality improvements and methodological enhancements resulting from information gained in the previous survey cycle.
|
From page 158... ...
With the urging of OMB to develop a web-based questionnaire, an opportunity exists to do cognitive testing and a codification of processing and editing rules. The panel urges SRS to take the lead in the work on the industrial survey, using the tools of the interagency agreement, the oversight of a high-quality methodological staff, and the input of highly qualified outside experts.
|
From page 159... ...
The panel further recommends that, over time, SRS develop both the internal staff capacity in data analysis and a suitable vehicle or vehicles for professional publication of data analysis by both internal staff and outside experts. MEASURES OF EFFECTIVENESS The discussion of the strengths and weaknesses of SRS has so far focused on the uniqueness of the agency, the organizational structure, and the principles and practices that must be in place for SRS to fulfill generally accepted minimum requisites of a federal statistical agency.
|
From page 160... ...
This 30-month period contrasts unfavorably with the timeliness experience for the industrial READ survey as reported by SRS for the previous 6 years, since the large-scale redesign of the survey between 1991 and 1992 (data displayed by reference calendar year in months from reference month to publication of full detail) : 993 1994 1995 1996 1997 2000 19 17 23 15 13 30 The data for the past few years for all of the major SRS surveys show that things have become worse, not better, since the previous NRC panel recommended improvement in trmeliness.
|
From page 161... ...
What is the entire globalization picture? There is some information on research in the United States funded by companies abroad and some information about research abroad funded by the United States, but there is no overall picture of the entire process.
|
From page 162... ...
SRS does not have the staff it needs to manage the Census Bureau staff well and to insist that the survey be done to meet well-accepted statistical standards. Balance The panel has observed that SRS, in the press to produce its surveys, cannot devote sufficient attention to broader concerns of data quality and coverage.
|
From page 163... ...
TOOLS TO IMPROVE DATA QUALIFY Quality starts with an organizational commitment and is buttressed by professional standards of practice. These are two recognized elements that help define an effective statistical agency (National Research Council, 2001b)
|
From page 164... ...
Other specific actions that have improved or will lead to Improvements include the addition of mathematical statistics expertise to the SRS senior staff, the incorporation of specific contractual obligations for measuring and presenting measures of quality in contracts with the data collection organizations responsible for conducting the several survey programs, and development of a long-term plan for methodological research for each program. Mathematical Statistics Expertise The 2000 NRC report focused considerable attention on perceived staffing issues in SRS, calling for enhancing staff expertise through professional development activities, augmenting staff expertise in statistical methods and subject-matter skills, and developing a more Interactive relationship with outside researchers.
|
From page 165... ...
Three of the research items were designed to address OMB clearance conditions: record-keeping and administrative records practices, effects of mandatory versus voluntary reporting on the state items, and web-based survey operations. Other items on the list of nine key activities include research on data collection below the company level, cognitive research on survey forms and instructions and identification of appropriate respondents, survey sample design issues, survey processing (data editing, cleaning, and imputation)
|
From page 166... ...
Another consequence of forcing an entry is that traditional measures of nonsampling error, such as item nonresponse and imputation rates, may be artificially low; statements in NSF publications that "item nonresponse is not viewed as a problem" because item nonresponse rates are under 1 percent may be quite misleading as a measure of overall quality. Against this backdrop, OMB has challenged NSF to explore the possibility of web-based collection in the larger and more stylized Survey of Industrial Research and Development.
|
From page 167... ...
These data quality improvements included (not surprisingly) a significant decrease in response variation in overlapping reference periods, a reduction in spurious reports of change, and less coding error and editrng work.
|
From page 168... ...
The federal funds and college and university surveys attempt to eliminate the problem of unit nonresponse completely, by seeking 100 percent compliance from the universe of reporters. Item nonresponse is actively discouraged, either by making it difficult not to enter an item in a web-based report, or by encouraging reporters to estimate information when actual data are not available, as is the practice on the federal funds report.
|
From page 169... ...
The reported values for item nonresponse rates for key data elements in the survey are also quite problematic. Imputation rates are published, but they are a poor proxy for item nonresponse rates.
|
From page 170... ...
· Research record-keeping practices of companies to determine if line-of-business data could be collected on the industry survey. Across-the-board improvements will be made only when NSF augments its own internal staff expertise in order to exercise greater control over the content and operations of the surveys—a process that has begun in the agency but on a limited basis.
|
From page 171... ...
The purpose of the workshop was to generate suggestions for improving measurement, data collection, and analysis. Workshop participants included statisticians and economists concerned with industrial organization and innovation practices, industrial managers, association representatives, government of ficials representing diverse policy arenas and statistical agencies, and analysts from international organizations and other industrialized countries.
|
From page 172... ...
Disk Disk Disk Disk Development 149,000 139,000 170,000 185,000 1.50 1.50 1.50 1.50 1997 93 100.0 Disk 226,000 1.50 1998 92 100.0 Web 267,000 1.50 1999 90 100.0 Web 209,000 1.50 2000 93 100.0 Web 315,000 1.50 2001 74 100.0 Web 170,000 1.50 2002 74 Web 379,000 1.75 2003 74 Web 334,000 1.75 Smvey of Federal Science and Engineering Snppott to Univetsities, Colleges, and Nonprofit Institutions 1993 15 100.0 Disk (sample size includes major agencies only)
|
From page 173... ...
2003 585 Web/paper 1,267,000 600,000 1.75 Survey of Research and Development Funding and Performance at Nonprofit institutions 1996 515,000 1997 9,112 41.4 Paper/web 515,000 NOTES: Research Facilities Su vey is a biennial su vey. FTE totals exclude the nonprofit su vey.
|
From page 174... ...
Both the Survey of Federal Funds for Research and Development and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit institutions benefit from respondent issue workshops. A formal special emphasis panel has not been established for these federal surveys.
|
From page 175... ...
For the Survey of Research and Development Fundrng and Performance by Nonprofit institutions, a special emphasis panel was established at the very outset to guide the development of survey content and sampling coverage. In 1996, this panel met to provide guidance and make suggestions for the planned 1996 and 1997 nonprofit ROD data collection effort.
|
From page 176... ...
The panel recommends increased reliance on mandatory reporting between economic censuses to improve data quality, reduce collection inefficiencies, and provide greater equity among reporters. However, the panel also recommends additional research on the topic of voluntary versus mandatory reporting, to investigate whether mandatory reporting is the most effective strategy (Recommendation 8.5)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.