Appendix V

Five- and Ten-Year Review Summaries and Panel Members

This appendix includes excerpts from internal reports to the National Academies Keck Futures Initiative (NAKFI) Oversight Committee developed during the work of the Five-Year and Ten-Year Panels.

FIVE-YEAR PANEL REPORT SUMMARY

After the first 5 years of the program, the NAKFI Five-Year Review Panel reviewed the founding assumptions of the program as well as each program element. The panel also recommended short-term modifications to strengthen the program and offered suggestions for ways in which the program could be expanded over the longer term. A summary of the Five--

TABLE V-1 Summary of the NAKFI Five-Year Review Panel Recommendations and Program Response

Year Review Panel’s recommendations and how NAKFI responded is shown in Table V-1.

Many of the recommendations and findings of the NAKFI Five-Year Review Panel were incorporated into the program. While the Five-Year Review Panel found that there was a misalignment between the program goals and the strategies, they also concluded that the program strategies should continue. The impact of the recommendations of the Five-Year Review Panel resulted in changes to the program, many of which continued through 2017.

NAKFI TEN-YEAR EVALUATION AND PLANNING PANEL REPORT SUMMARY

The charge of the NAKFI Ten-Year Evaluation and Planning Panel was to:

• Formulate recommendations for how to logically wind down the last few years of the program;
• Identify potential opportunities for continuing NAKFI’s mission of advancing interdisciplinary-related research and institutional goals; and
• Develop suggestions for how the NAKFI surplus fund might be used in this context.

The work of the NAKFI Ten-Year Evaluation and Planning Panel included a review of what the program had accomplished since the Five-Year Review and a scan of how the environment for interdisciplinary, transformative, and innovative research had changed in the past 10 years. It was determined that concept mapping was a well-documented and validated approach for accomplishing the environmental scan. The concept mapping approach is a rigorous, mixed methods one, blending qualitative idea generation and sorting and rating those ideas with several multivariate analyses. Concept mapping is an efficient way to construct shared conceptualizations of complex issues; it invites diverse perspectives into idea generation, sorting, and ranking. The results can be used for a variety of purposes including needs assessment, project design, program evaluation, and strategic planning.

The first stage of the process is to decide on a carefully worded focus statement for online idea generation. In order to solicit a broad set of ideas

including, but not limited to, interdisciplinary research, the focusing statement for this project was “Research would be more innovative if….” Members of the NAKFI network, comprised of more than 1,500 people, were invited to a secure website where they were asked to complete the sentence, “Research would be more innovative if….” The website allowed respondents to see previously submitted ideas, search for keywords, and contribute ideas over multiple sessions. A total of 910 statements were generated by 494 participants.

The panel reduced the ideas to a manageable number by choosing 88 statements that represented the ideas of the larger group. Sorting the ideas required participants to have the experience and background knowledge to make connections among the ideas and to judge which ideas belonged together by meaning or topic. Approximately 150 people familiar with the NAKFI program were invited to assist with the sorting activity. Included in this group were members of the W. M. Keck Foundation staff and board, previous Futures Conference Steering Committee chairs and committee members, Futures Grantees, select members of the Communication Award Selection Committee, and others. Concurrently, all members of the NAKFI network were asked to participate in the rating task, which involved rating each idea on two separate, 5-point Likert scales on importance and on the extent to which the participant believed the idea was commonly practiced. Approximately 38% of the NAKFI network participated in one or more aspects of the concept mapping project.

The core of the concept mapping process is developing a collective view of the relationships among the ideas. This involves three steps:

1. Create a similarity matrix from sort data. The similarity matrix is a square symmetrical matrix that shows the number of participants who sorted each pair of statements together in accomplishing their sorts.
2. Multidimensional scaling (MDS) of similarity matrix, to locate each statement as a separate point on a two dimensional (X, Y) map (i.e., point map).
3. Hierarchical cluster analysis of the multidimensional scaling (X, Y) coordinates to partition the points (statements) on this map into groups.¹

___________________

¹ Kane, M., & Trochim, W. M. K. (2007). Concept Mapping for Planning and Evaluation. Thousand Oaks, CA: Sage.

For this project a combination of computerized solutions and human sense making determined the final number of groups in the concept map (see Figure V-1). Computing the concept map required multidimensional scaling and hierarchical cluster analysis to integrate the sorting information from each individual, converting that qualitative information to quantitative data and developing a series of easily readable concept maps and reports.

Following the development of the initial concept map, each member of the NAKFI Ten-Year Evaluation and Planning Panel was interviewed about the map’s structure and content. Panel members were asked to respond to the content of the map, provide suggestions for how to label the clusters, and what the map might mean for the evaluation of NAKFI and the program’s future. The final solution for NAKFI’s concept map on innovative research contained 10 clusters of ideas (see Figure V-1). Each numbered dot represents an idea. Ideas that were close together were grouped together more frequently by participants.

The labels for each cluster were determined by the panel through consensus. The panel discussed how the ideas fared on the importance and common practice ratings. Ideas considered to be both high in importance and not currently practiced were especially valuable to informing the evaluation of NAKFI and generating recommendations for the future.

Key findings from the ratings data included:

• Clusters rated with the highest importance were Responsive Funding, Venture Science, and Ingenuity & Risk-Taking—all areas where NAKFI excelled.
• Academics’ perceptions of what is important are very similar to non-academics.
• Perceptions of what is important are very similar for all participant groups by age, field, and gender.
• Academics and non-academics agreed that the clusters Responsive Funding, Venture Science, Ingenuity & Risk-Taking, Incentives, Collaborations, and Engagement were not currently practiced or were rarely practiced.
• Participants’ perceptions of what is important were divergent from their perceptions of what is currently practiced.

Based on the 88 ideas generated from this process the panel settled on a multidimensional scaling solution of 10 groups, which were further simplified to 7 broader areas (see Figure V-2). The panel also examined how ideas and groups of ideas fared on the importance and current practice ratings. Finally, the panel considered how the findings might impact the future of NAKFI and developed recommendations.

As shown in Figure V-2, the model for creating more innovative research includes familiar areas: institutional support; responsive funding; ingenuity and risk-taking; incentives; education and training; and data accessibility. Interdisciplinary collaborations form the center of the model. Each of the 88 statements connects to 1 of the 7 areas in the model. Table V-2 displays sample ideas and their associated areas. This project provided a consensus-based approach to priority setting, program development, and evaluation that could be applied to NAKFI as well as other programs or organizations.

Many of the areas in the model were leverage points NAKFI had already successfully engaged. The program directly addressed the core issue of interdisciplinary collaborations by drawing together researchers across science, engineering, and medicine, as well as members of media, government organizations, foundations, and the general public and providing them the space, time, and funding needed to explore and act on new solutions to intractable intellectual challenges. For graduate students and post-doctoral scholars involved in a Futures Grant, their participation provided direct experience conducting interdisciplinary research. According to the NAKFI

alumni who participated in this project, “more funding” was not the only answer to the question of how to support more innovative research. A variety of funding mechanisms are called for, including support for proof-of-concept studies, small awards that do not require preliminary data, and longer-term funding for especially promising young investigators. NAKFI’s flexible seed grants succeed in this area.

Some of the areas were not ones that NAKFI or the National Academies could address directly. For example, NAKFI was unlikely to be able to influence the policies and infrastructure support of other institutions. However, the National Academies could play a role in providing consensus-based recommendations to higher education around innovative promotion and tenure guidelines that support and facilitate cross-disciplinary research collaboration. Data accessibility/sharing is a complex issue that could be developed into a potential conference theme, drawing together professionals with expertise in information science, data privacy, computer science, visualization, and related fields.

TABLE V-2 Sample Ideas Contained in Each Area of the Model

Participants were asked to rate along two dimensions: Importance (how important each idea is relative to the other ideas; 1 = not at all important to 5 = extremely important) and Current Practice (the extent to which each idea is currently practiced relative to the other ideas; 1 = never practiced to 5 = very often practiced). This scan of the landscape for how to elicit more innovative research provided insight into the relative importance of these areas and the extent to which these ideas are commonly practiced. Figure V-3 displays the Importance by Current Practice rating averages by area. Participants’ rankings were consistent across fields (science, engineering, and medicine), sectors (academia versus non-academia), ages, and genders.

The 2014 environmental scan identified that work remained to be done. Not one of the areas ranked high in importance was also viewed as being commonly practiced. Many of the areas in which NAKFI excelled (Funding, Venture Science, and Ingenuity & Risk-taking) were considered to be of high importance and not commonly practiced (lower right), thus supporting the panel’s conclusion that NAKFI provided a unique and valuable model for fostering cross-professional research. Creating incentives for innovative research is the domain of institutions of higher education and not an area where NAKFI could or should have been expected to have a direct impact.

NAKFI FIVE-YEAR REVIEW PANEL MEMBERS

James J. Duderstadt (NAE) (Chair)

President Emeritus and University Professor of Science and Engineering

University of Michigan

Floyd E. Bloom (NAS/IOM)

Professor Emeritus

Department of Molecular & Integrative Neuroscience

The Scripps Research Institute

Enriqueta C. Bond (IOM)

President

Burroughs Wellcome Fund

Robert W. Conn (NAE)

Managing Director

Enterprise Partners Venture Capital

Timothy L. Killeen (NAE)

Director

National Center for Atmospheric Research

Neal F. Lane

University Professor and Senior Fellow

James A. Baker III Institute for Public Policy

Rice University

Cherry A. Murray (NAS/NAE)

Deputy Director for Science and Technology

Lawrence Livermore National Laboratory

Gilbert S. Omenn (IOM)

Professor of Medicine, Genetics & Public Health

Center for Computational Medicine and Biology

University of Michigan Medical School

Anne C. Petersen (IOM)

Deputy Director

Center for Advanced Study in the Behavioral Sciences

Stanford University

Adele J. Wolfson

Professor of Chemistry & Associate Dean Wellesley College

Department of Chemistry

National Academies Staff

Anne Heberger

Evaluation Research Associate

National Academies Keck Futures Initiative

Kenneth R. Fulton

Executive Director

National Academy of Sciences

External Consultants

Alan L. Porter

Evaluation Coordinator and Professor Emeritus of Public Policy;

Co-director, Technology Policy & Assessment Center

Georgia Institute of Technology

David Roessner

Sr. Evaluation Consultant and

Associate Program Director

Science and Technology Policy Program

SRI International

NAKFI TEN-YEAR EVALUATION AND PLANNING PANEL MEMBERS

Terrance Sejnowski (NAS/NAE/IOM) (Chair)

Investigator, Howard Hughes Medical Institute;

Francis Crick Professor

Salk Institute for Biological Studies

Enriqueta Bond (IOM)

Founding Partner

QE Philanthropic Advisors

Former President, Burroughs Wellcome Fund

Robert Nerem (NAE/IOM)

Institute Professor and

Parker H. Petit Professor Emeritus

Institute for Bioengineering and Bioscience

Georgia Institute of Technology

Anne C. Petersen (IOM)

Research Professor

Center for Human Growth and Development

University of Michigan

Founder and President, Global Philanthropy Alliance

Thomas A. Schwandt

Professor, Quantitative and Evaluative Research

Methodologies and Chair

Educational Psychology Department

University of Illinois at Urbana-Champaign

National Academies Staff

Kenneth R. Fulton

Executive Director

National Academy of Sciences

Anne Heberger Marino

Program Officer

National Academies Keck Futures Initiative

Kimberly Suda-Blake

Senior Program Director

National Academies Keck Futures Initiative

External Consultants

Marie Cope

Client Services Consultant

Concept Systems, Inc.

Scott Rosas

Senior Consultant

Concept Systems, Inc.

This page intentionally left blank.