6
Summary of Key Themes from the Workshop
The nonprofit sector is an important and growing sector in the U.S. economy, as many participants noted during the workshop. Yet, many widely used existing datasets, such as the System of National Accounts and the National Patterns of R&D Resources, capture data in ways that mask the true scope and nature of the sector. Moreover, the research and development (R&D) activities in a portion of this dynamic sector have not been directly measured in 18 years.
This workshop brought together experts in economics, survey methodology, and nonprofit sector research, as well as representatives from different nonprofit organizations and the National Science Foundation (NSF), to discuss the design of a new survey of R&D in the nonprofit sector. Through presentations and moderated discussions, participants considered the nature of the nonprofit sector and the funding landscape, the ways in which R&D can vary in this sector, and the challenges these differences pose. They also examined in some detail approaches to designing the sample with available sources of data, as well as strategies for measuring, implementing, and developing outputs from the survey.
Framing these broad topics was the need that NSF expressed to provide data on the “other nonprofit” sector for national reporting on R&D across all sectors. These data are published annually in the National Patterns of R&D Resources and are ultimately used for international comparisons. For this reason, decisions and definitions that facilitate “apples to apples” comparisons are an important consideration.
Over the course of the workshop, the topic of definitions was a theme that infused each session. Multiple participants raised the issue of the narrow way that the Frascati Manual defines the nonprofit sector, and they suggested that NSF weigh the benefits and drawbacks of adhering strictly to that approach. The definitions and terminology associated with research, development, and R&D as a unified concept were addressed multiple times. The definition of work itself was raised as another issue by several participants. Ultimately, they said, decisions on these definitions will have significant implications for sampling, measurement, analysis, and reporting.
A related theme that many participants discussed at length was the unique nature of R&D in the nonprofit sector. Representatives from six nonprofit organizations who presented at the workshop described a wide range of activities ranging from basic and applied research to educational and social service innovations, among others, using various terms. Many indicated these types of activities differ from the traditional definitions of R&D, but are important to capture. Discussion also revealed the complex relationships that exist between umbrella organizations and their subsidiaries and the partnerships that exist between nonprofits and other organizations such as universities. These relationships not only raise the risk of double-counting an organization’s activities, participants noted, but also make it challenging to identify a single respondent knowledgeable about the organization’s R&D activities, particularly at the larger institutions.
Methodological considerations flow from the conceptual choices that NSF will make about the survey, several presenters pointed out. With a priority placed on gathering data for national reporting, a number of speakers addressed ways to maximize the efficiency of the sample given budgetary limitations. Several panelists suggested not only increasing the overall sample size from the 1996–1997 sample size, but also using stratification and weighting techniques to yield a sample with greater numbers of large producers of R&D than of smaller organizations less likely to produce R&D. However, others also suggested a variety of flexible techniques for ensuring the representation of smaller organizations and more variable uses of the data in the future. Some data to inform these approaches are available from the Internal Revenue Service and the National Center for Charitable Statistics. Screening approaches and the use of auxiliary data are other ways to maximize efficiency, boost response rate, and aid later analysis of the data, according to several presenters.
Many presenters and participants repeatedly voiced a need to move away from the questionnaire format of the previous survey from 1996–1997, noting its lengthy and complex definitions and outdated categories. Instead, they suggested ways to develop screening tools that deconstruct complex concepts into multiple-question formats, informed by recent
research on this subject. The rapidly advancing science of survey design also provides specific steps that survey implementers could take to obtain good response rates. In particular, survey methodologists who spoke during the workshop emphasized gathering extensive contact information and planning a synergistic series of contacts during implementation.
The need for data that are compatible with those of other existing surveys and international standards creates a set of statistical and methodological demands that may be at odds with capturing data that represent the full diversity of the nonprofit sector given limited resources, according to a number of participants. However, presentations and discussions addressed ways in which NSF could build in flexibility for future data needs and/or establish partnerships to enable broader research goals to accompany NSF’s specific goals for the survey. Suggestions included collaborating with other government agencies to provide a clearer view of the nonprofit sector or developing partnerships with external researchers or foundations that can conduct complementary research on a wider range of nonprofit organizations.
Finally, establishing ongoing communication with nonprofit organizations emerged as a theme expressed by many. These communications, including the ongoing exploratory interviews that NSF is conducting and highlighted during a session, can yield important information about how the nonprofit sector describes its own R&D activities, providing valuable insight for survey design. However, fostering this dialogue can also facilitate shared understandings about the value of the sector and the survey for NSF, the nonprofit sector, and for the nation as a whole.
Lester Salamon suggested some themes from the workshop on behalf of the steering committee. He offered support for moving forward with a two-track approach. The first track would involve meeting NSF’s need for accurate data on the nonprofit sector to satisfy its aggregate reporting needs, and the second track would involve taking steps “to understand the underlying reality” of the nonprofit sector. Salamon expanded on what he sees as key issues for each of these tracks.
MEETING NSF’S NEED FOR DATA
Pursuing the first track begins with clarifying the types of R&D activities that NSF is planning to capture. In Salamon’s view, NSF’s need to gather current data about R&D in the nonprofit sector to improve its reporting across sectors has opened an opportunity to clarify and expand the existing definition of R&D. He commended NSF for its willingness to think in new ways about these activities, as well as the ways in which the nonprofit sector itself has been evolving. In particular, service and expressive organizations are increasingly using evidence-based decision-making
processes that entail engaging in data collection and research. This type of activity “needs to be reflected in any conception of R&D going forward,” stated Salamon. In summary, he said, older conceptions of R&D need to be broadened.
The complex relationships among organizations in the nonprofit sector will also have to be addressed to ensure that NSF obtains useful data, Salamon stated. Research in the nonprofit sector can be intramural or extramural, undertaken by multiple entities and/or their larger umbrella groups, and/or be a shared enterprise by various types of organizations. Because of these relationships, NSF will need to take steps to avoid double-counting nonprofit organizations.
Volunteers are an important segment of labor among nonprofit organizations, and NSF will need to determine how to measure the value of their work, Salamon indicated. He added international groups of labor statisticians have considered changes to the definition of work to include volunteer labor as a form of work, ultimately including it in the system of national accounts, signaling a major shift in thinking about work. These shifts would redefine gross domestic product, and he suggested that future measurement of work, including R&D, be informed by these decisions.
The terminology that the survey uses to elicit participation and to identify R&D producers is another central issue that surfaced in the discussion, according to Salamon. He suggested that the term “research” should probably not be used at all on the survey because it can lead to a circular definition. Instead, he said, terms such as “data gathering,” “systematically analyzing the data,” or “applying the results of data analysis to practice” are potential ways of describing research. Salamon noted, “I’m sure we can’t in this version drop the term ‘research’ altogether, but at least NSF may need to formulate a set of words that it would use to explain what it means by ‘research’ in the survey.”
Identifying the correct respondent within a nonprofit organization able to respond to questions about R&D is another issue that Salamon spoke about. Presentations from Donald Dillman, Jeffrey Berry, and others provided particular concrete steps that could facilitate this process and increase the likelihood of a response to the survey, he added. Salamon also encouraged NSF to include organizations similar to the Hillside Family of Agencies in ongoing exploratory interviews. Contacting umbrella groups associated with various social service sectors is one way to identify such representative organizations, followed by asking leaders in these umbrella groups to identify their members who may be engaging in R&D or evidence-based decision making. Salamon stated, “I couldn’t agree more with the notion of getting a good [sample] frame, but I think the good frame has to be driven by the definition, by the concept.”
Finally, for NSF to achieve its goals for the survey with a satisfactory response rate, the survey must have a good value proposition for respondents. Salamon indicated that NSF will need to put the survey in context for organizations in terms that will resonate for the organizations that devote time to responding.
UNDERSTANDING THE NATURE OF R&D IN THE FULL NONPROFIT SECTOR
Salamon described a second track devoted to developing a greater understanding of the full range of R&D activities in the nonprofit sector, including the full breadth of various types of organizations and the nature of their R&D activities. To begin to accomplish this larger goal, he suggested conducting in-depth, qualitative case studies of 30 to 40 exemplary organizations engaging in some form of research or development. The case studies would serve as a means to analyze the range of activities and the words being used to describe them. They would also help in determining which of these R&D activities would be desirable to include or exclude for various future purposes.
This qualitative research would facilitate future research efforts that build on the NSF survey that will already have been implemented. In Salamon’s view, this additional research would expand the sample and add a broader body of data on the nonprofit sector that is a more accurate reflection of its diversity. He added this work could be external to NSF with data sharing or in partnership with NSF, but argued that NSF should “not let this drop and assume that whatever it is that NSF can do and will do will be sufficient to really allow us to capture the full reality of nonprofit research and development activity.” Based on discussions at this workshop, nonprofit and philanthropic communities appear to be in the midst of a significant new appreciation of the importance of evidence-based decision making. Salamon stated that in his opinion it would be unfortunate if a major study of R&D in the nonprofit sector failed to acknowledge this important trend.
NSF SUMMARY OF WORKSHOP
In closing the workshop, John Gawalt, director of NCSES at NSF, offered his perspective on the origin of the survey and the lessons learned from the workshop. Despite the challenge of initiating a new survey that must be used in conjunction with other surveys with specific needs and an existing history, he said NCSES found support and recognition of the need to conduct a new survey of R&D in the “other nonprofit” sector. Thus, NCSES received funding to address this recognized gap in the
data; however, the nature of the funding cycle is responsible for the rapid timeline for preparing the survey for implementation. He noted that steps that would typically be done sequentially are being completed in parallel.
NSF sought to conduct the present workshop to gain insight and advice on the design and implementation of the survey. Gawalt added that although NSF will not be able to follow through with all suggestions made at the workshop, it has gained information to inform its current choices and to explore potential future partnerships. He indicated that the immediate next steps will involve engaging in a process of discernment and discussion, taking a reasoned approach to making the necessary policy and methodological decisions. No decisions have been reached on the frequency with which the survey of R&D in the nonprofit sector will be conducted going forward.
Gawalt noted that the expertise marshaled by the Committee on National Statistics and the passion and engagement of the workshop participants has benefitted the process and furthered his understanding of the diversity of the nonprofit sector. Furthermore, direct engagement with representatives of the sector, the potential respondents, was also valuable as NCSES considers the design of the survey. In particular, Gawalt said he will focus on balancing the needs of NCSES and the broader sector, communicating about the importance of the sector to various constituencies and improving outreach to the nonprofit sector. He asked the representatives of nonprofit organizations for their help in developing messages and outreach approaches that will resonate with organizations and demonstrate the value proposition of the survey.
Gawalt expressed interest in pursuing paths to facilitate a true understanding of the sector. One reason is that the nonprofit sector is growing at a time when other sectors are contracting. Building in flexibility to add contextual information, future phases, and new ideas would be desirable, he said. In addition, he expressed interest in learning how other larger government statistical agencies, such as the Census Bureau and the Bureau of Labor Statistics, are engaging with the nonprofit sector and what they have learned. Gawalt stated that the workshop had “achieved one thing, which is to take my thinking from ‘I need a survey with a set of numbers’ to ‘I need to be able to explain or describe the nonprofit sector and how research fits in and how that sector contributes.’” He said the workshop has broadened his thinking about the sector, future analyses, and ways to highlight the nonprofit sector on a national stage.
