National Academies Press: OpenBook
« Previous: 7 Models for Institutionalizing Experimentation in U.S. Federal Agencies
Suggested Citation:"8 Closing Roundtable." National Academies of Sciences, Engineering, and Medicine. 2024. Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27244.
×

8

Closing Roundtable

In a brief final roundtable discussion, five participants—Heidi Williams (Stanford University), Jon Lorsch (National Institutes of Health), Alan Tomkins (National Science Foundation [NSF]), Caleb Watney (Institute for Progress), and Adam Jaffe (Brandeis University)—shared some of the major lessons they took away from the workshop.

Williams, who moderated the session, began by reiterating, as an example, the request by Diana Epstein (Office of Management and Budget) for a toolkit that would facilitate arranging for employees under the Intergovernmental Personnel Act (see Chapter 6). “Things like that that [provide] a to-do list for people who are trying to make things in the field work better,” she said.

Watney cited his interest in comments by Hoangmai Pham (Institute for Exceptional Care) on the launch of the Center for Medicare and Medicaid Innovation (CMMI) and how its structure and circumstances contributed to its successes and shortcomings. Such information “really matters both for setting up future organizations and for getting organizations right in the future.” Enlisting someone to do a deep oral history of the early stages of CMMI could provide a more complete picture.

Lorsch urged science and innovation policy researchers to incorporate some of the ethical thinking done in the world of clinical trials into their randomized controlled trials (RCTs). For example, ethical precepts could help ensure that researchers do no harm relative to a standard of care. “In the context of giving grants, issuing patents, or giving reimbursements for medical services,” Lorsch said, “the bar to doing an RCT should be extremely high because the standard of care produces some value.” Randomizing someone to a particular group could produce a worse outcome, “in which case the taxpayers are getting a worse service, and you are certainly going to hear about it.” If someone does not get a grant, a patent, or a reimbursement, people in the randomized group could attribute that outcome to being in the randomized group. He continued, “I would urge you to look at clinical trial ethics and try to superimpose that on what it is you think should be done in terms of government experiments.”

Suggested Citation:"8 Closing Roundtable." National Academies of Sciences, Engineering, and Medicine. 2024. Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27244.
×

Lorsch also said that at some point, depending on how much money is involved, an experiment becomes a program, “and an old saying in government is that it’s very easy to start programs but very hard to end programs. Once you get them going, they take on a life of their own.” The instigators of experiments need some way to stop a program if the evidence shows that the program is not beneficial, he said.

Finally, Lorsch pointed out that “not every idea for an experiment is worth doing.” Just as thought needs to be devoted to what scientific experiments will give taxpayers returns on their investment, experiments in the policy sphere need to be considered in that light.

Citing the need for systematic inquiries, Tomkins called for more funding for research on the science of science. NSF has such a program, but other agencies should support this research as well. “It’s another area of scientific inquiry, and it’s one that is undervalued and underexplored,” said Tomkins. As a specific example, more research on public access to results and open science could help reveal the benefits and costs of this approach.

Jaffe said that he was struck by the statements that small amounts of money spent on evaluation and understanding can have very large impacts on the effectiveness of large expenditures. He also pointed to the value of collaborations that can enable work in agencies that do not have enough personnel to do much experimentation.

Finally, to close the workshop, Jaffe pointed out the major goal of figuring out “ways to help people move from trying new things to building a culture of systematic learning about what works better and what doesn’t work as well.” This requires a cultural change, which cannot happen quickly. “But I was always taught that the perfect shouldn’t be the enemy of the good,” he said. “We should keep trying to do better than we were doing before. The more that we specifically and systematically organize ourselves to be learning, the more we’ll learn.”

Suggested Citation:"8 Closing Roundtable." National Academies of Sciences, Engineering, and Medicine. 2024. Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27244.
×
Page 69
Suggested Citation:"8 Closing Roundtable." National Academies of Sciences, Engineering, and Medicine. 2024. Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27244.
×
Page 70
Next: Appendixes »
Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop Get This Book
×
 Experimental Approaches to Improving Research Funding Programs: Proceedings of a Workshop
Buy Paperback | $23.00 Buy Ebook | $18.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On March 14-15, 2023, the Board on Science, Technology, and Economic Policy of the National Academies of Sciences, Engineering, and Medicine held a 2-day workshop in Washington, DC to explore the use of data, research, and experiments to improve the processes for and outcomes of federal funding of scientific research. The workshop brought together researchers in the science of science funding and practitioners from government and the private sector with experience supporting or carrying out experimentation and evaluation to discuss illustrative examples of the use of experimentation from the United States and abroad; consider methods of evaluation; and foster relationships for future experimentation.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!