Skip to main content

Currently Skimming:

APPENDIX C: THE PROMISES AND LIMITATIONS OF PERFORMANCE MEASURES--Irwin Feller
Pages 119-152

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 119...
... Its meanings and implementation can vary from forcing fundamental changes in the ways in which public sector organizations and assessed and thus public funds allocated, as evinced by recent state government initiatives across all levels of U.S. education, to constituting old wine in new bottles, especially to empirically oriented economists, 119
From page 120...
... A stylized dichotomization is as follows: • endorsement of requirements for, belief in, scholarly search supportive of, and opportunistic provision of performance measures that respond or cater to executive and legislative branch expectations or hopes that such measures will facilitate evidence-based decision making; • research and experientially based assessment that even when well done and used by adepts, performance measures at best provide limited guidance for future expenditure decisions and at worst are rife with potential for incorrect, faddish, chimerical, and counterproductive decisions. The tensions created by these differences are best captured by the observation of Grover Cleveland, 22d and 24th President of the United States: "It's a condition we confront -- not a theory." The condition is the set of Congressional and Executive requirements upon Federal agencies to specify performance goals and to provide evidence, preferably in quantitative form, that advances towards these goals have been made.
From page 121...
... . The second branch is the accumulated and emerging theoretical and empirical body of knowledge on the dynamics of scientific inquiry and the processes and channels by which public sector support of research produces societal impacts.
From page 122...
... Section IV illustrates the specific forms of the promises and limitations of performance measures in the context of what it terms the "big" and "small" questions in contemporary U.S. science policy.
From page 123...
... . Likewise omitted are consideration of the normative goals underlying Federal support of research and the distributive effects or societal impacts that flow from it (Bozeman and Sarewitz, 2011)
From page 124...
... Plentiful too and continuously being updated are compendia and manuals covering international best practice on how to evaluate public sector R and D programs. These works cover a wide range of performance impact measures and methodologies, including benefit-cost analysis, patent analysis, network analysis, bibliometrics, historical tracings, innovation and on the outputs produced by several different Federal agencies -- health, energy, agriculture, environmental protection, international competitiveness, employment.
From page 125...
... Performance Impact Measures Differences in assessments about the potential positive and negative features of requiring strategic plans and performance measures into how Federal agencies set research priorities and assessed performance were visible at the time of GPRA's enactment. They continue to this day.1 In 1994, almost immediately after GPRA's passage, I organized a session at the American Association for the Advancement of Science's (AAAS)
From page 126...
... • Well defined, readily measured, and easily communicated performance measures aids both funders and performers to communicate the accomplishments and contributions of the public investments to larger constituencies, thereby maintaining and
From page 127...
... • Performance measurement focuses attention on the end objectives of public policy, on what has happened or happening outside the black box, rather than on the churning of processes and relationships inside the black box. This interior churning produces intermediate outputs and outcomes (e.g., papers, patents)
From page 128...
... The largest promise of performance measurement, though, likely arises not from recitation of the maxims of the new public management but from the intellectual ferment now underway in developing new and improved data on the internal processes of scientific and technological research, the interrelationships of variables within the black box, and improved methods for assembling, distilling and presenting data. Much of this ferment, of course, relates to Dr.
From page 129...
... Limitations The above noted emphasis on context surfaces immediately in considering the limitations of performance measurement. Perhaps the most obvious and important difference in the use of such measures in
From page 130...
... If retrospective assessment was all that was implied by the call for performance measures of impacts, the task before this audience, and for Federal science agencies in satisfying new planning and reporting requirements, while challenging, especially in reconciling and distilling divergent, at times conflicting findings, as say in the cases of the BayhDole Act (Larsen, 2010; NRC, 2010) or the SBIR program's generation of sustainable increases in employment, would at least be relatively straightforward.
From page 131...
... . The primary limitation of using performance measures to shape future Federal investments in research flows from the well documented tale, widely recounted in both the scholarly and policy literatures, that the outcomes of scientific research are unpredictable as to when they will occur, who will be responsible for them, and even more so with respect to their end uses.
From page 132...
... For example, although suffused with an emphasis on quantitative performance measures, OMB's earlier articulation of R and D Investment Criteria expressed nuanced understanding of the uncertainties surrounding returns from Federal investments in basic research: "Agencies should define appropriate output and outcome measures for all R and D programs, but agencies should not expect fundamental basic research in the same way that applied research or development are able to do. Highlighting the results of basic research is important, but it should not come at the expense of risk-taking and innovation" (OMB, PART 2008, Appendix C, p.
From page 133...
... The limitations of performance measures as forms of evidence to guide investments in research extend beyond this general case. There are other specific limitations that arise in or bear upon specific decisions in specific contexts.
From page 134...
... • In a circular process, unless a program's objective is defined in terms of a single performance measure, any single measure is at best a partial indicator of the objective being pursued. In most cases, single performance measures are only loosely connected to higher order performance objectives.
From page 135...
... performance in Nobel prizes to health outcomes produces only a slightly better "picture." The mapping of Nobel prizes in physiology and medicine for the period 1945-1984 with health statistics shows a slight positive relationship in reductions in infant mortality but no apparent association to gains in life expectancy. Compare this use of this single performance measure with Cutler and Kadiyala's estimate that an average 45-year old in 1994 had a life expectancy 4 ½ years longer than in 1950 because cardiovascular disease mortality had decreased.
From page 136...
... An obvious implication of the vignettes from health research is the risk of using any single performance measure to gauge the impacts of Federal investments in research. This proposition is stated so frequently and explicitly in contemporary exegesis on assessment of science programs (Schmoch, et.
From page 137...
... The potential consequences for misinterpreting evidence and subsequent questionable decisions when effectiveness and efficiency are confounded takes on special importance in light of the recent OMB memorandum, "Increased Emphasis on Program Evaluations." The memorandum states that "Rigorous, independent program evaluations can be a key resource in determining whether government programs are achieving their intended outcomes as well as possible and at the lowest possible cost." It also notes that, "And Federal programs have rarely evaluated multiple approaches to the same problem with the goal of identifying which ones are most effective." Absent some form or control or comparison group or other explicit standard, performance measures provide little basis for determining a program's cost-effectiveness or efficiency. • The informational content of performance measures may change over time.
From page 138...
... It also provides the intellectual and policy capital base for consideration of a continuing flow of smaller questions and smaller answers about specific science and innovation policy issues. These latter issues flare up to dominate near-term science policy forums, and then through some amalgam of a modicum of resolution, overtaking by the eruption of new policy agenda items, or by morphing into the big questions lose their immediate saliency, only to
From page 139...
... the allocation of these investments among missions/agencies/and programs and thus fields of science; and (3) the selection of performers, funding mechanisms, and the criteria used to select projects and performers.4 Permeating each of these questions is the question of "why," namely, the appropriate, effective and efficient role of the Federal government in supporting public investments in science and technology including nurturance of a STEM-qualified labor force.
From page 140...
... They were initially advanced to justify public sector support for basic research but as interpreted in the 2002 OSTPOMB R and D Investment Criteria and then implemented in the PART process, they have become the theoretical basis for excising several domestic technology development programs.
From page 141...
... Given all the above cited reservations about the complexity of linking public sector research expenditures to desired outcomes, how can performance measures be used to exist to judge the merit of recent proposals that the U.S. should be spending 6 %, not 3% of GDP on R and D (Zakaria, 2010)
From page 142...
... Exciting as it may be to envision the prospects of societal impacts flowing from frontier, highrisk, transformative risk, it serves only to bring one full circle back to the policymaker's priority setting and resource allocation questions noted above. The same issues arise when trying to compute the proper level of support or estimate the returns to public investments for functional objectives, agencies, and fields of science.
From page 143...
... The challenge at this point is not the absence of performance measures relating Federal investments in research to specific outputs or studies pointing to high social rates of return within functional areas but the sheer number of them and the variations in methodologies that produce them. The result is a portfolio of options about performance measures, each more precisely calibrated over time but still requiring the decision maker to set priorities among end objectives.
From page 144...
... Referring to the number of independent studies working from different models and different data bases that have pointed to very higher social rates of return, he noted, "But it is evident that these studies can provide very limited guidance to the Office of Management and Budget or to the Congress regarding many pressing issues. Because they are retrospective, they shed little light on current resource allocation decisions, since these decisions depend on the benefits and costs of proposed projects, not those completed in the past" (Mansfield, 1991, p.
From page 145...
... A different category of benefits owing less to improved public sector management practices and more to the realities of science policy decision making needs to be added to this list. The very same arguments cited above that the links between initial Federal investments in research are too long term and circuitous to precisely specify in GPRA or OMB planning or budget formats serves to increase the value for intermediate measures.
From page 146...
... Finally, as an empirically oriented economist whose work at various times has involved generating original data series of patents and publications and use of a goodly portion of the performance measures and methodologies now in vogue in evaluations of Federal and State science and technology programs, there is a sense of déjà vu to much of the debate about the promises and limitations of performance measures of impacts. The temptation is to observe somewhat like Monsieur Jordain in Moliere's play, Le Bourgeois Gentilhomme, "Good heavens!
From page 147...
... It is to (1) have policymakers employ performance impact measures that correspond to what is known or being learned about how public investments in basic and applied science relate to the attainment of given societal objectives; (2)
From page 148...
... Mapping the Frontiers of Evaluation of Public Sector R and D Programs.
From page 149...
... Young. National Bureau of Economic Research Working Paper 16300.
From page 150...
... 2005. Science, Technology, and Innovation Policy.
From page 151...
... 1998. Effective Use and Misuse of Performance Measurement.
From page 152...
... 2009. Empirical Observations on New Public Management to Increase Efficiency in Public Research-Boon or Bane?


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.