Skip to main content

Currently Skimming:

5 Methods of Assessing Science
Pages 95-123

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 95...
... As Table 5-1 shows, various assessment methodologies have arisen from different disciplinary and multidisciplinary perspectives, measuring different aspects of scientific activity, and addressing various science and technology policy and assessment questions.
From page 96...
... For example, the National Science and Technology Council (1996:xii) set forth the following nine principles for assessment of fundamental science programs: • Begin with a clearly defined statement of program goals.
From page 97...
... Understanding Risk noted that science policy decisions typically employ both analysis and deliberation and argued that it is appropriate for them to do so. Among the reasons identified for using deliberation are that the most useful type of analysis often is not self-evident and is best determined through dialogue involving both the potential producers and the users of the analysis, and that judgment is inevitably involved in finding the meaning of analytic findings and uncertainties for specific decisions, particularly when the decisions must be made against multiple objectives.
From page 98...
... in research policy making. Thus, it is useful to focus attention on a set of empirical questions such as these: • Can deliberations about the past progress of scientific fields and the best way to shape research portfolios be better informed by the use of appropriate analytic methods?
From page 99...
... Finally, we turn to analytic-deliberative approaches. A familiar one in the context of the NIH is the Consensus Development Conference, which combines analysis and deliberation but has not been adapted for making research policy decisions.
From page 100...
... We consider the potential roles of analytic techniques in that light. Bibliometric Analysis The term scientometrics broadly relates to the generation, analysis, and interpretation of quantitative measures of science and technology.
From page 101...
... One potentially valuable contribution of bibliometrics to the assessment of scientific fields is that it makes possible the assessment of the import and export of ideas between fields by following cross-citation patterns (van Leeuwen and Tijssen, 2000)
From page 102...
... The pilot study strongly suggested that if bibliometric indicators are to be used for research assessment, considerable reliance must be placed on the subject-matter experts to guide and review the work of the specialists in bibliometrics who will perform the actual studies. Several iterations of generation and analysis of data will probably be needed before the assigned experts are satisfied with the output.
From page 103...
... Decision analysis, by contrast, provides a set of techniques that can be used to organize and structure deliberation. Decision-analytic techniques have not been given much attention in science policy, and, when proposed as decision aids, they have often met stiff resistance from scientists (Fischhoff, 2000; Arkes, 2003)
From page 104...
... Qualitative techniques of decision analysis, by which we mean techniques for structuring or organizing decision problems without attempting to quantify the effects of decisions, are more modest in their objectives than the quantitative approaches, but they seem to have greater potential for assisting with priority setting in research policy. Decision analysis, used to structure choices, can make decision processes more transparent, thus contributing to accountability, by creating frameworks for examining issues, focusing deliberation on explicit evaluative criteria, and helping diverse groups understand the bases of their divergent judgments (North and Renn, 2005)
From page 105...
... suggests that the process of developing influence diagrams of the pathways from research to its scientific results and societal benefits can clarify the place of various research activities in the larger enterprise and promote more focused discussion of priorities, even if credible numbers cannot be calculated to estimate the strengths of the relationships that the arrows represent. Such discussion could systematically address such questions as whether anyone in the scientific community is receiving research support to understand each element in the influence diagram and "whether the research investments are com mensurate with the opportunities" (Fischhoff, 2000:82)
From page 106...
... Thus, it began by identifying factors that are likely to act as means to the ends implied by each criterion. For example, the panel members agreed to rate potential science priorities highly on the criterion of likelihood of achieving significant scientific advances if the following factors were judged to apply (p.
From page 107...
... Any such measures will need considerable development and validation if they are to become useful, and, regardless of how much validation work is done, we emphasize that measures of the dimensions of scientific progress should be used as inputs to deliberative processes, not as replacements for them. This is because priority setting inevitably requires a weighing of the various dimensions of progress and the various program goals.
From page 108...
... They typically meet in person and, through discussion, arrive at collective judgments about the quality of research proposals or programs that are used as advisory input by research managers. Peer review panels typically do not rely in any explicit way on scientometric or other analytical methods (see Bornmann and Daniel, 2005)
From page 109...
... Although the copious literature on peer review contains numerous examples and personal observations on how the system operates in specific cases, we have found no systematic research on peer review for making comparisons among scientific fields, a type of assessment that raises issues somewhat different from those central for assessing individual research proposals. Thus, research on peer review processes must be interpreted carefully to draw conclusions about how such processes might best be used for comparing fields.
From page 110...
... . We have found no comparative studies of peer review processes for priority setting that examine how the structure or process of their deliberations affects the results.
From page 111...
... Thus, deliberative processes that include both producers and the various kinds of users or beneficiaries of projected research deserve serious consideration by BSR in setting research directions for areas that are suspected of needing improvement in terms of their production of useful knowledge. ANALYTIC-DELIBERATIVE METHODS Analytic-deliberative methods are those in which judgment is based in part on information from scientific theory and data or other systematic sources of knowledge.
From page 112...
... . Thus, in neither field is it easy to find expert reviewers who are competent across the range of substantive areas to be reviewed, and in neither field is there a satisfactory common metric for comparing research progress.
From page 113...
... Knowledge benefits include not only knowledge that is applicable to technology or health care, but also improved basic understanding of processes of aging even if that knowledge has no foreseeable application. Prospective analyses would involve judgments of the likelihood that research investments would yield knowledge, options, and realized benefits of the types desired by BSR and NIA.
From page 114...
... NIH Consensus Development Conference Model The Consensus Development Conference is a familiar analyticdeliberative process in NIH. This model, which has been used more than 120 times since 1977, follows a carefully thought out rationale and set of procedures (for a detailed description, see http://consensus.nih.gov/ ABOUTCDP.htm)
From page 115...
... They are chaired by "a knowledgeable and prestigious person in the field of medical science under consideration" who is not "identified with an advocacy position on the conference topic." They include research investigators in the field, health professionals who use the technology in question, methodologists, and "public representatives, such as ethicists, lawyers, theologians, economists, public interest group or voluntary health association representatives, consumers, and patients." Members are selected for their ability to weigh evidence and to do collaborative work, as well as for their absence of identification with advocacy positions or financial interest related to the conference topic. The consensus development model has been used in NIH for providing advice on a variety of policy-related topics, but not in the area of research policy.
From page 116...
... As these examples and experience in other areas of public policy decision making suggest, processes that incorporate relevant analytic techniques and information into deliberations in groups that represent the range of scientific knowledge and policy perspectives needed for wise decisions can result in recommendations and decisions with several desirable properties. The recommendations and decisions can be well informed about the available evidence, systematic in consideration of the evidence from all relevant policy perspectives, accountable, and even consensual among groups representing diverse perspectives.
From page 117...
... We consider it possible to constitute expert review panels that draw on their own experiences and insights, augmented by quantitative data on the outputs, outcomes, impacts, productivity, or quality of research, to arrive at better informed and more systematically considered expert judgments about the progress and prospects of scientific fields than they could reach without quantitative data. We think that processes that organize ongoing exchanges of judgments between bodies of scientists and science managers can produce wiser decisions than processes based on either-or thinking.
From page 118...
... Despite the existence of a considerable body of historical case research, little systematic knowledge exists about the paths of the development of science, particularly behavioral and social science with applications to health and well-being; about the roles of government agency decisions in that progress; about the possibility of accurately measuring and assessing such progress; about the best ways to use analytic approaches to improve decision making; or about the best ways to structure decision making to take advantage of information from studies of science. Research Needs Several lines of research can contribute to the knowledge base needed for a social science of science policy (Marburger, 2005)
From page 119...
... • Analyses of research vitality or interest shown by active scientists in lines of research, focusing on research directions that are widely considered in hindsight to have been successful or unsuccessful in terms of yielding major scientific advances or societal impacts. The studies should examine the ways that the vitality of scientific fields may relate to subsequent scientific outcomes and impacts.
From page 120...
... These studies might compare the consequences for the development of scientific fields, particularly new fields, of research portfolios that emphasize large centers, database development efforts, or interactive workshops, with more traditional research portfolios emphasizing funding to individual investigators and small research groups. Research on the roles of science agency decisions in scientific progress can help the offices and agencies that sponsor it to make decisions about how to select and train research managers and organize advisory groups so as to better promote program goals for advancing science.
From page 121...
... • Studies to assess the value of providing information developed through specific analytic techniques, such as bibliometric studies, for research priority setting. Studies using cross-citation patterns or analyses of academic and professional career trajectories of researchers and students can show whether such analyses add significantly to the decision-relevant knowledge of expert review groups and whether and how this information alters their recommendations.
From page 122...
... • Studies to adapt existing analytic-deliberative assessment approaches, such as the NIH Consensus Development Conference model to the purposes of assessment of research areas and research priority setting in BSR. Some of these studies might incorporate the above techniques for informing and structuring decisions.
From page 123...
... • Comparative studies of advisory panels of different composition, particularly for recommending research priorities. For example, BSR, NIA, and the NIH Center for Scientific Review might vary the breadth of expertise of experts or the balance between senior and junior researchers.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.