5
The Next Generation of
Researchers and Practitioners
The three actors central to advancing and applying the research framework now outlined are (1) established scholars in the fields and specialties identified in Chapter 4; (2) Ph.D. candidates in those fields and specialties; and (3) administrators and faculty responsible for curricula in schools and programs summarized below by the term “policy education.” For the first two of these actors, there are historical and contemporary models we briefly note; the third will involve fresh thinking.
In 1923, the Social Science Research Council (SSRC) was established to promote “co-operative research among the several disciplines” (Fosdick, 1952, p. 198),1 as a necessary foundation for creating entirely new research fields and specialties. Later the term “field development” was coined. We use that term to describe a coordinated and well-funded effort to attract established scholars to important but under-researched issues for which their theories and methods are appropriate.
Citing metaphors favored in that more naïve time, the director of the Laura Spelman Rockefeller Memorial, Beardsly Ruml, who funneled millions of dollars to research universities and the SSRC in the 1920s, lamented that “All who work toward the general end of social welfare are embarrassed
________________________
1This early plea for interdisciplinary research did not use the term, which did not appear (as interdiscipline inquiries) until SSRC’s Sixth Annual Report, 1929-1930 (noted by Sills, 1986).
by the lack of that knowledge which the social sciences must provide.” Ruml offered what for him was the clinching argument (cited in Fosdick, 1952, p. 194):
It is as though engineers were at work without an adequate development of physics and chemistry, or as though physicians were practicing in the absence of the medical sciences. The direction of work in the social field is largely controlled by tradition, inspiration and expediency.
Ruml and the SSRC leadership had a clear goal: to professionalize the social sciences, provide them methodological tools necessary for rigorous research, and point them toward important fields of investigation.
Ruml was not naïve about the challenges: data were meager; research was based on second-hand observations and anecdotal material; classroom instruction isolated students from social conditions; and, especially, the social sciences were challenged to investigate topics that could not “be brought into the laboratory for study,” but “must be observed if, when, and as operative.” Difficulties notwithstanding, “unless means are found for meeting the complex social problems that are so rapidly developing, our increasing control of physical forces may be increasingly destructive of human values” (cited in Fosdick, 1952, p. 195).
We bring this early philanthropic initiative to mind to draw a lesson still applicable. Targeted funds can help develop new research specialties. The well-funded SSRC emphasized interdisciplinary research and a strong commitment to empirical methods. Social science researchers responded not only to the SSRC, but also to the program priorities announced by other philanthropic foundations, to the Russell Sage Foundation in more than a century of social science funding and to the larger foundations—Ford, Carnegie, Hewlett, and MacArthur, among others—in the second half of the 20th century. The label field development, for example, was attached to area studies, a Cold War era success story. Coordinated conferences, workshops, research monographs, and edited volumes advanced research focused on showing how recently decolonized countries could engage in “nation building” and how western democracies should meet the threat of Communism, which in turn spawned a generation of research on the Soviet Union and China watchers (so-called because lacking access to the Chinese mainland, they “watched” from Hong Kong). There are many examples of new research fields promoted by private foundations and government
funders—behavioral economics, human dimensions of climate change, population studies, life course development, aging, and race, ethnic, and gender studies.
An important example directly related to understanding the use of social science across a broad array of public policies is the ambitious effort pioneered by neoconservative social scientists who were skeptical about the effectiveness of many Great Society programs. With the Olin Foundation in the lead, private funds subsidized books, endowed university professorships, offered student fellowships, and established think tanks—similar to strategies earlier pioneered by the Spelman Rockefeller Memorial Fund—“all with the intent of changing the prevailing terms of debate” and advancing market-sympathetic policy (Rodgers, 2011, p. 7). This effort shaped the ongoing debates about the respective merits of the state and the market with respect to a long list of social policies—including whether poverty was better reduced by social welfare government programs or market forces, and whether school reform was better advanced through vouchers and school choice than leaving many educational practices under the influence of teacher unions.
Field development is not limited to foundations, though they have been particularly adept at it. The U.S. National Science Foundation (NSF) is attentive to how its funds might shape fields of inquiry, noting that in addition to its reliance on a well-established peer review process to guide its grant making, program officers should “identify promising research that responds to national priorities identified by Congress and the Administration” and to “incorporate agency or programmatic priorities” in NSF funding (Marrett, 2011, p. 3).2 Particularly important to our purpose, the portfolio of grants funded by the NSF is expected to achieve “special program objectives and initiatives” and to build “capacity in a new and promising research area” (Marrett, 2011, p. 5).
A highly visible and in some quarters sharply criticized foray is the recent Science of Science Policy Initiative (Fealing et al., 2011). The broad purpose is to develop an evidentiary base for policy decisions on investments in basic and applied scientific research. New federal programs associated with this purpose include the Science of Science and Innovation Policy at NSF and an interagency task force sponsored by the National Science and Technology Council. A virtual community of practice has been organized,
________________________
2For a technical description of NSF’s merit review policy, see http://www.nsf.gov/bfa/dias/policy/meritreview/ [February 2012].
facilitated by the establishment of a website hosted by the Office of Science and Technology Policy in the Executive Office of the President.3 We cite this example not to endorse it (for a critical analysis, see Feller, 2011) but to illustrate how federal funding is used to develop new fields of inquiry: in this example, it is an attempt to build a community of practice among researchers and between researchers and policy makers.
Two sponsors of our study, the William T. Grant Foundation and the Spencer Foundation, have specifically targeted funding to better understand the use of research in policy and practice with respect to children and youth (W.T. Grant) and data and information to improve education (Spencer). The W.T. Grant Foundation sponsors research on the acquisition, interpretation, and use of research evidence to develop “strong theory and empirical evidence on when, how, and under what conditions research is used.” Its request for proposals notes that “[r]esearch acquisition, interpretation, and use occurs within a social ecology” and that the foundation seeks “to understand how organizational, social, economic, and political contexts matter” (William T. Grant Foundation, 2012).4
The Spencer Foundation’s Evidence for the Classroom Project, part of its broader Data Use and Educational Improvement Initiative, sponsors research on the assumptions behind data-based educational reforms “by investigating whether, when, and how student performance data informs instruction in K-8 classrooms.” The goal is “to learn more about how K-8 teachers use student performance data for instructional decisions and how organizational and individual factors affect that use.” Included in this initiative is research on how organizations learn and improve (Spencer Foundation, 2012).
In a review of Spencer-funded research papers published in the American Journal of Education, Goren (2012) notes that the papers “call for a deeper and better understanding of data, their use, the conditions that are most conducive for using data well, how individuals and groups of practitioners make sense of the data before them, and the intended and unintended consequences of data use for school improvement” (p. 233). The summary conclusion laments “that our understanding of how data lead to improvement in education is tremendously underdeveloped” (p. 234).
These Grant and Spencer examples are consonant with the research agenda described in Chapter 4. As valuable as they are, however, they
________________________
3For details, see http://www.scienceofsciencepolicy.net [February 2012].
4For a description of this research program and early lessons from its funded research, see Tseng (2012) and the accompanying commentary.
touch on a small subset of the issues that need study in order to develop a deeper and wider understanding of the use of science in the policy context. The Grant Foundation initiative is limited to children and youth, and the Spencer Foundation initiative is limited to data use as a particular feature of educational practice. Similarly, the NSF example noted above is limited to science policy, and it is narrow in its selection of research methods.
If these initiatives are joined by sponsored research on how science is used as evidence in many other policy areas—international security, economic growth, renewable energy, transportation efficiency, agricultural productivity, etc.—and are targeted to methods and approaches described in Chapter 4, a new research field on the scale of area studies or behavioral economics would take shape. Of course, established scholars have already worked out their future research, and we cannot expect more than a small percent to shift their interests to the framework in Chapter 4 (though we welcome being proven wrong). Science funders have long accepted this reality, and have often focused on entry-level researchers as better candidates for launching new fields and specialties. With this in mind, we turn next to Ph.D. candidates.
PH.D. TRAINING: AN ENTRY POINT
A well-tested strategy for establishing new research fields provides incentives early in a person’s research career, especially at the dissertation phase. The SSRC pioneered this approach in the 1920s, eventually offering hundreds of fellowships in the social sciences that produced leaders in the academically based departments and in the steadily growing array of policy institutions (Fosdick, 1952, pp. 230-231). Another major chapter in the history of philanthropic leadership was the substantial, decades-long funding of graduate training in languages and area studies by the Ford and Mellon Foundations, in service of enlightened foreign policy. The 1958 National Defense Education Act (Title VI) added federal funds to this effort.
In more recent decades, dissertation grants provided by the MacArthur Foundation added depth to international security education, successfully reorienting the field from a 1960s focus on a limited array of issues, primarily arms control, to a broader consideration of how international economics, global immigration, climate change, and other “nonsecurity” issues were, in fact, deeply implicated in how the nation should approach its security challenges in the 21st century. The predoctoral research training program in the neurosciences, sponsored by the National Institutes of Health (NIH),
encouraged broad, early-stage training in the neurosciences. This program was targeted to basic and disease-related research of importance to the participating institutes.5 A successful current effort is NSF’s Integrative Graduate Education and Research Traineeship (IGERT) Program. Initiated in 1997, the program is intended “to establish new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries” (National Science Foundation, 2012c).
These funding initiatives, and there are others, have in common a determination to establish new fields by starting with researchers in the earliest stages of their professional training. The strategy rests on a simple assumption. Ph.D. candidates searching for a dissertation topic are attracted to new areas, where a single study can quickly be influential. The dissertation is the basis for their early publications, which, if cited, keeps them on this track. Enough young scholars on a similar track begin to establish a new field. This time-tested strategy fits with a central point of this report: attracting a fresh generation of researchers to studies of the use of science in policy should not be difficult in this period of heightened political (and, we expect, funder) attention to whether the substantial public investment in science—social sciences included—results in science that is used. The list of research topics is long—this is a small sample:
• Challenges in linking the natural and social sciences in the policy context;
• How variability in the quality of scientific evidence affects its use;
• The role of intermediaries in promoting evidence use;
• The responsiveness of policy makers to commissioned research;
• The interaction of scientific claims and value claims in policy argument; and
• Comparative research that considers how different government systems produce and use scientific evidence for policy and how this relates to differing political systems and beliefs about the role of government.
Based on their disciplinary training—in systems analysis, studies of complex organizations, science and technology studies, social psychology,
________________________
5For details, see http://grants.nih.gov/grants/guide/pa-fles/PAR-00-037.html [February 2012].
behavioral economics, political science, statistics, cognitive sciences, and the history of science—Ph.D. candidates can start with the substantial research literature on how scientific knowledge is produced and proceed quickly to what is not known about how science is used as evidence in policy making, and then apply methods and theories, already available from their disciplinary training, best suited to remedying the gaps in knowledge. These beginning scholars are guaranteed two attentive audiences for their work. There is an influential audience of public and private science funders, government agencies, institutes, think tanks, lobbyists, and others with a stake in whether relevant scientific knowledge is brought to bear in policy. The second audience is faculty responsible for what is being taught to students en route to careers in the policy enterprise, to which we now turn.
POLICY EDUCATION: WHAT IS NEEDED
Training beyond the bachelor’s degree is a minimum job requirement for almost all public policy positions. Perhaps mentoring and on-the-job learning worked in an earlier period, when policy challenges slowly made their way to the public agenda and arrived as fairly straightforward questions of whether X leads to Y. That world, if it ever really existed, is clearly not today’s policy world. A nation dependent on policy analysts and policy makers who learn as they go is put at risk when policy challenges (as well as information, both helpful and unhelpful) arrive at bewildering speed, from unexpected directions, and in ever more complex forms. Professional preparation is the norm today, and university-based programs are where that preparation occurs.
Senior policy positions often require (or assume) Ph.D.-level training, but a significant number of positions in the policy enterprise recruit from programs leading to a master of public administration (M.P.A.), the degree traditionally offered in schools of public policy, though now more likely to be labeled master of public policy (M.P.P.). This relabeling reflects the shift from careers in the civil service to those in the policy enterprise. Related training takes place in other professional schools, especially law, business, public health, social work, and education. There are also programs focused on particular policy arenas, such as environmental policy, security policy, and urban policy. Some of these have become stand-alone master’s degrees, an increasing practice in higher education (Radin, 2000). Although the United States leads the world in establishing public policy schools and programs, similar initiatives are now found on every continent and in steadily
growing numbers. We use the generic term “policy education” to cover the M.P.A., M.P.P., topical master’s degrees, and related certificate programs.
This array of programs presents an obvious entry point for introducing fresh ways of thinking among those who will practice policy analysis and program design. Their education should be based on two priorities. One is now being taught—acquiring the competencies relevant to assessing policy-relevant research knowledge. One is not—developing a clear understanding of the factors that influence the conditions under which that knowledge is likely to be used.
These joint priorities distinguish policy education from what is provided in academic departments, where the priority is primarily the discovery of new knowledge—even recognizing that academic social scientists increasingly hope that their research will be used. Policy education also differs from what aspiring political consultants and policy advocates seek (though many looking for such careers earn a M.P.A. or M.P.P.), which are skills relevant to advancing a political cause or winning a policy battle. The academic social sciences adequately attend to the education of advanced students whose vocation is the discovery and dissemination of knowledge. The political world adequately supplies on-the-job training for those whose vocation is winning through bargaining and compromising, media campaigns, mobilization of support, and using science evidence selectively and tactically. Neither the academically oriented nor the politically motivated student is the audience we have in mind. Rather, it is the student whose priority is bringing scientific evidence to bear on policy choices, and wanting this not for tactical reasons but because it is a core professional principle. As Majone (1989, p. 7) writes:
The job of analysts consists in large part of producing evidence and arguments to be used in the course of public debate. Its crucial argumentative aspect is what distinguishes policy analysis from academic social science on the one hand, and from problem solving methodologies such as operations research on the other. The arguments that analysts produce may be more or less technical, more or less sophisticated, but they must persuade if they are to be taken seriously in the forums of public deliberation.
The statement of task guiding this report did not direct the committee to conduct a comprehensive investigation of what is being taught in policy programs and schools. Deliberations of the committee, however, led to the
firm belief that it is timely to examine policy education in the same spirit that the famed Flexner Report (Flexner, 1910) examined medical education a century ago and the Ford Foundation (Gordon and Howell, 1959) and the Carnegie Corporation of New York (Pierson, 1959) examined business education in the 1950s. The Flexner Report, commissioned in 1908, stands out in this list; it is widely credited with initiating reforms that professionalized medical and health training appropriate for 20th century challenges, and from which the nation continues to benefit.
An analogous effort directed to policy education could determine if schools and programs are suitably aligned with the challenges that have emerged over the past half-century: decolonization; democratization; globalization; mass communication and the emergence of the Internet; economic and technological development; the international diffusion of science and technology; the rise of knowledge elites; and the growing influence of the private sector in information production and knowledge management, in addition to the host of specific competencies associated with evidence-based policy, performance metrics, cost-benefit analysis, and evaluation research. A Flexner-like effort could determine whether policy schools are providing the knowledge and skills relevant to assuring that policies responding to these broad challenges are influenced by science.
In the absence of such a study, we turn to a research literature offering partial though important insights into policy school objectives and the implementation of those objectives. In addition, the committee conducted its own cursory examination of the curricula of nearly 100 policy schools and programs in the United States. We acknowledge that what is readily available allows only best-guess estimates about what is being taught every year to the thousands of students enrolled in public policy courses. Although we would prefer to have a Flexner-like exhaustive study at hand, our immediate question can be adequately answered with what is available: how much of what we endorse as a policy education curriculum is already in place?
We are confident that practically all public policy education includes courses on the “politics of policy making.” These courses draw on a large political science literature that examines how political considerations affect policy outcomes. There is also attention to the role of values, a topic appearing in any number of topical courses on the assumption that value tradeoffs appear in practically all policy choices. Examples include intergenerational choices, such as abundant energy for current generations versus the risk of sea-level rise that will inundate coastal communities of future generations; allocating public funds between competing public goods, such as repairing
roads versus lower student-teacher ratios; deciding who should pay for policy failures, whether the costs of the collapse in the housing market should be borne by those who borrowed above their means or by those who packaged the mortgages in ways that hid the risks. More generally, students are taught that the complexity in policy making results not just from weighing counterarguments about effectiveness and efficiency, but also from facing questions about what is right, just, or fair.
If political and value considerations are being routinely taught, so are methods. In these courses there is a decided emphasis on quantitative skills. Morçöl and Ivanova (2010) document this, and categorize the quantitative methods courses into three groups: (1) research design courses, in which experimental and quasi-experimental designs are favored; (2) data collection methods, in which surveys are favored; and (3) analytic approaches, in which regression analysis is favored. That is, it is clear that methods associated with the “evidence-based policy” framework (see Chapter 3) are strongly represented in policy education curriculum. This is to be expected, and policy education should continue to emphasize the quantitative methods relevant to analyzing social conditions, designing responsive policy interventions, and evaluating the consequences of interventions.
However, as detailed in Chapter 4, other competencies are needed to navigate the policy world. These competencies include attention to the properties of reasoning about scientific knowledge (Grozer, 2009) and to understanding the assumptions underlying divergent policy framings, expert judgments, consensus-building techniques, and analytic methods or approaches. This knowledge will help prepare students to cope with the realistic, everyday problems encountered in applying existing knowledge—with its gaps, imperfections, and disciplinary constraints—to policy problems. Without such understanding, students may overestimate the persuasive power of scientific reasoning, and overlook the substantial barriers of institutional and cultural resistance to new research knowledge, unfamiliar policy framings, or solutions that challenge deeply held moral or ethical beliefs. Internships and case studies can help students learn about these and other complexities of the policy-making process.
Because the case study method is widely used in policy education, we reviewed a large number of case studies from the perspective of our report. Consistent with the observation above, cases used in policy schools routinely cover how political considerations influence policy outcomes and value tradeoffs. They draw student attention to the distribution of benefits and costs and how the “rules of the game” condition policy choices. They
use to advantage a large number of key concepts and processes—from bureaucratic inertia to unintended consequences, from negotiation strategies to using the media.
What the examined cases rarely attend to is how scientific knowledge is used in policy making. There is little discussion of the quality or quantity of research available to the policy makers, even less discussion of whether that research is used as evidence, and still less about why science is ignored. Except incidentally, the cases do not explore the role of knowledge brokers or whether the ideas of evidence-based policy come into play. The processes and institutions through which policy makers gain access to relevant knowledge, such as expert advisory committees, receive little notice. There certainly is no attention to whether variation in cognitive biases of policy makers or variation in cultures of decision making tell them what to expect when science enters the policy argument. In summary, practically nothing of what is emphasized in Chapter 4 as ways to better understand the use of science is reflected in the case studies we examined.
An additional suggestive finding comes from Great Britain, where the current government has established the Behavioral Insights Team, a small office led by a social psychologist. Thaler (2012) describes how this office used a randomized control trial to test behavioral theory on when people conform to social norms. The issue was tax compliance; the treatment was a letter to late payers stating that others in their community pay their taxes promptly. There was a sharp increase in compliance in the treatment group, and not in the control group, whose message made no mention of neighbor’s behavior. British tax authorities estimate that the reinforcing message could generate extra annual revenue of £30 million ($46.5m) nationwide. We cite this small study because the government (Thaler, 2012, p. 4) “is sufficiently convinced of the value of these activities [of the Behavioral Insights Team] that it announced last week that behavioral science is to be included in the required curriculum for civil servants.” Behavioral science had not been taught in Britain’s civil service training but now will be.
Though it would take a Flexner-style investigation to offer a thorough account of what is today being taught to thousands of M.P.A. and M.P.P. students in U.S. universities, our cursory review points to what is absent. Our review found few courses that draw on social psychology and cognitive science to provide public policy students with an understanding of human decision-making processes—including biases, heuristics, and probabilistic errors—as they pertain to reasoning about policy. Nor did we find many courses in which an anthropological, sociological, or humanistic approach
to policy making is used to help students make sense of the interconnectedness of actors and institutions and the frameworks that shape policy choices. Nor did we find policy education to be self-conscious about the issue one might expect it to be most attentive to: what do students need to understand about the use of scientific evidence in public policy?
The social sciences have the opportunity to influence the competencies and perspectives that today’s students in master’s-level policy programs carry with them into positions across the policy enterprise. We hope that this report will spur self-examination across policy schools. One outcome might be differentiation, with some programs providing ever more rigorous training in methods and theories that strengthen research about “what works” and other programs emphasizing rigorous training in methods and theories that strengthen understanding of the conditions needed to put that research to policy use. Such a division of labor would result in a broad array of perspectives and skills available to think tanks, legislative staffs, policy units in executive branches, and other settings in the policy enterprise—from local government to international agencies, in both the public and the private sector.
There is no better way to summarize this chapter than repeating a truism—effective public policy is dependent on a steady supply of well prepared graduates prepared for public service and associated careers in the policy enterprise. Our report advocates a broad definition of well prepared, certainly to include technical competencies in evaluation research, program design, measurement, and the like—but to include as well an understanding of how science can be used to inform public policy.
The committee writes this report mindful that the American public’s willingness to invest in science education and research is not unlimited, and that the immediate times emphasize scrutiny of the investment. But these times are also witness to a steadily growing policy enterprise—a broad effort to make “better” policy through the application of science. We have not taken a position on “better” policy, but have certainly taken a position on the value of, to return to our title, Using Science as Evidence in Public Policy. Moreover, we have written that it is within the competency of and is therefore an obligation of the social sciences to advance our understanding of “using science.”
