Evaluation of the Minerva Research Initiative (2020) / Chapter Skim
Currently Skimming:
4 Research Supported by the Minerva Program: Quantity and Quality
4 Research Supported by the Minerva Program: Quantity and Quality
Pages 61-78
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 61... ...
documents and interviews with DoD staff, the grantee survey, the sponsored research administrators survey, discussions with national security experts and other stakeholders at public committee meetings, and the Minerva Conference. The committee also analyzed the outputs reported by the Minerva grantees.
|
From page 62... ...
Accordingly, the committee's review of the outputs of Minerva-sponsored research involved compiling lists of the outputs, coding and summarizing the outputs by type and subtype, and examining in greater depth the journals in which articles were published and the citations of those articles by other researchers as an indicator of contributions to the social science knowledge base. The committee also considered outreach and dissemination to nonacademic audiences by reviewing nonacademic publications reported by Minerva principal investigators (PIs)
|
From page 63... ...
Additionally, rules for which types of articles are peer-reviewed vary across journals. For example, the website of Nature Research states that the journal subjects articles, letters, brief communications, technical reports, analyses, resources, reviews, perspectives, and insight articles to peer review, while the website of the journal International Organization states that it sends all submissions, except letters to the editor, to reviewers.
|
From page 64... ...
should ensure that the analyses and conclusions concerning the outputs from Minerva grants are reasonably robust. Columns 2 and 3 of Table 4-1 show the numbers of publications and presentations produced by Minerva grantees by type, and for each type, the proportion of the total reported by the 57 PIs who provided this information.
|
From page 65... ...
. Whereas Table 4-1 provides summary data for all publications and presentations reported by all responding PIs, Table 4-2 provides summaries, by year of the PI's grant (2009 to 2017)
|
From page 66... ...
Journal impact metrics also have been used inappropriately to assess individual researchers even though they were developed as journallevel indicators. Although all journal-level impact metrics have limitations, two with relatively strong measurement properties are CiteScore and the Scimago Journal Ranking (SJR)
|
From page 67... ...
2 In some cases, similar fields with only one journal were combined to form larger subject field groups. For example, the following eight subject fields that each have one or two journals were combined into one collapsed subject field called computer science, engineering, and mathematics: computer graphics and design, computer science application, general computer science, media technology, control and optimization, control and systems engineering, electrical and electronic engineering, and applied mathematics.
|
From page 68... ...
Of the articles in that field, 55 percent were published in journals ranked in the top CiteScore decile, while 89 percent and 91 percent, respectively, were published in journals ranked in the top quartile according to CiteScore and SJR. The table also reveals considerable variety in the subject fields of the journals in which articles produced by Minerva-supported grantees have been published -- consistent with the intent of the Minerva Research Initiative to support interdisciplinary research.
|
From page 69... ...
9 1 11 9 100 1 11 All Fields 138 63 46 114 83 106 77 NOTE: SJR = Scimago Journal Ranking. or authoritativeness)
|
From page 70... ...
counts of the number of citations for every article published by a Minerva grantee; (2) "benchmarked percentiles that show how citations received by each document compared with the average for similar documents, relative to a certain subject field" (Scopus, 2019b)
|
From page 71... ...
bColumn total ÷ 125. TABLE 4-6 Comparison of Article Impacts and Journal Rankings for Minerva-Supported Articles Number of Articles Journal-Level CiteScore Percentile Article-level Row Benchmarking Percentile 90–99 75–89 50–74 30–49 <30 Total Percentagea 90–99 18 4 2 0 0 24 29 75–89 17 13 1 0 0 31 37 50–74 4 9 3 0 0 16 19 30–49 6 1 5 0 0 12 15 Total 45 27 11 0 0 83 100 Column Percentageb 54 33 13 0 0 100 Number Not Assigned a Benchmarking Percentile 18 24 7 3 3 55 aRow total ÷ 83.
|
From page 72... ...
Outreach and Dissemination to Nonacademic Audiences As noted in the earlier section summarizing the publications and presentations reported by PIs, in addition to 152 peer-reviewed academic publications and 582 presentations, Minerva PIs reported 333 other publications that included non-peer-reviewed journal articles, working papers, paper series, research briefs, commentaries and op-ed pieces, blogs, and newsletters. At least some of these publications appear to represent outreach and dissemination efforts aimed at nonacademic audiences to build public understanding of Minerva-supported research and highlight its policy implications.
|
From page 73... ...
RESEARCH SUPPORTED BY THE MINERVA PROGRAM 73 BOX 4-1 Other Products Developed through the Minerva Research Initiative Software • Analytic software based on social network analysis, involving natural language processing, geospatial analysis, and machine learning • Publicly available software for tracking online social movements, with applications for different countries Mapping Tools • A mapping tool that allows users to visualize data on climate change vulnerability, conflict, and aid to assess how climate change impacts and responses intersect • An online, interactive mapping tool providing policy makers with a platform to visualize trends in climate-related disaster vulnerability, conflict, gover nance challenges, and disaster aid • A tool for mapping extremist and counterextremist online narratives Websites • Content on complex emergencies and political stability • Content on climate change and political stability • Content on cyber international relations • Cyber-related content and sustainable development • A website mapping militants and militant groups • Websites covering transportation networks, land cover, settlements, land rights, land use, land tenure, toponyms, and population Databases • An events-based dataset on social conflicts • A dataset on armed conflict locations and events • A geocoded climate dataset • Education and infrastructure access data • A constitutional design database • A database on interstate and civil war termination • A dataset on autocratic regimes • Data on personalism in dictatorships • A database on maritime piracy events and locations • Climate and disaster aid data • Disaster response training data • Major episodes of mass nonviolent, violent, and "mixed" contentions continued
|
From page 74... ...
74 EVALUATION OF THE MINERVA RESEARCH INITIATIVE BOX 4-1 Continued Patents and Licenses • A visual intelligence platform for tracking online social movements • A system and method for contextual analysis • Methods for determining the similarity of content and structuring unstruc tured content from heterogeneous sources • Systems and methods for narrative and frame detection using generalized concepts and relations Training Materials • Climate change and development • Institutional capacity and natural disasters • Constitutional reform for conflict management • Climate change and national security • Languages across cultures Models and Methodologies • A climate security vulnerability model combining data on physical, socio economic, demographic, and political insecurities • A climate disaster aid transparency codebook • A climate security vulnerability model that identifies subnational locations most vulnerable to climate-related hazards, defined in terms of the potential for large-scale loss of life • A topic modeling evaluation tool • A language cohesion social media tool • A sensitive data encoder • A visual intelligence platform for tracking online social movements, with potential applications to programming more effectively, detecting and characterizing adversarial influence operations, analyzing the tactics of propaganda and disinformation, and tracking and refuting disinformation • An algorithm to inform a K-means algorithm for prototype move selection in a simulation STAKEHOLDER AND EXPERT PERCEPTIONS OF THE QUALITY OF MINERVA RESEARCH In addition to its analysis of the outputs generated by the Minerva grants, the committee reached out to a broad range of stakeholders and national security experts and solicited input on the Minerva program through a public comment mechanism in an effort to better understand perceptions
|
From page 75... ...
There appears to be broad agreement among both DoD staff and external stakeholders who provided input at the committee's public meetings that the Minerva Research Initiative is a unique program, and that the Minerva grants have attracted some top scholars and produced some highquality research. In recent years, the program has struggled with staffing challenges, which have resulted in delayed postings of the grant announcement and a corresponding decline in the number of white papers submitted (the number of white papers was 313 in 2016, 261 in 2017, and 192 in 2018)
|
From page 76... ...
. Considering the unique contributions of the Minerva program, national security experts remarked that the program has been successful in facilitating interdisciplinary research.
|
From page 77... ...
SUMMARY AND CONCLUSIONS The committee's ability to evaluate systematically all the products of the Minerva grants over the years was limited, and there was no group that could have served as a valid hypothetical comparison group for the volume and quality of the outputs produced by the Minerva grantees. It is clear, however, that the Minerva program has supported research that has been published in top journals and articles that have been cited much more than the average expected of their respective social science fields.
|
From page 78... ...
However, the input the committee received from national security experts, professional associations, and DoD staff indicated that the research is not as widely disseminated or utilized as it could be. Further discussion of this issue and the committee's recommendations for addressing it are included in Chapters 3 and 5.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.