Proceedings of a Workshop

The Role of Inducement Prizes

Proceedings of a Workshop—in Brief

INTRODUCTION

On May 29, 2019, the Committee on the Role of Inducement Prizes at the U.S. National Academies of Sciences, Engineering, and Medicine (the National Academies), in cooperation with the Laboratory for Innovation Science at Harvard (LISH), convened a workshop in Washington, D.C. on the role of inducement prizes to spur American innovation. Unlike prizes that recognize past achievements, these inducement prizes are designed to stimulate innovative activity, whether it be the creation of a desired technology, orienting research efforts toward designing products that are capable of being used at scale by customers, or developing products with wide societal benefits. Through a series of meetings, including this public workshop, the committee continues to gather evidence on the effectiveness of inducement prizes and how that can best be measured; how these prizes compare to grants and contracts; the broader benefits, beyond technology development, of holding prize competitions; the characteristics of prize winners and competitors; and how prizes fit into the broader spectrum of federal support for innovation.¹

As a key source of information for the committee, the workshop convened experts to address how prizes fit into federal and non-federal support for innovation, the benefits and disadvantages of prizes, and the differences between cash and non-cash prizes. Other discussion topics included the conditions under which prizes are most effective, how to measure the effectiveness of prizes, and the characteristics of prize winners. The workshop discussions are summarized below.

The workshop began with a welcome from committee chair, Karim Lakhani, of Harvard Business School. Dr. Lakhani first provided a historical perspective on prizes. Use of inducement prizes by governments has a long history. Examples include the City of Florence’s competition for a design for Il Duomo, the prize offered by the British Parliament early in the 18th century for an instrument to accurately determine longitude at sea, and Napoleon and the French Parliament’s prize for food preservation methods for French soldiers.

The internet has stimulated greater interest in prize competitions by lowering the cost of conducting contests and democratizing access to platforms themselves. Recent interest in prizes may be traced to the 1996 announcement by the XPRIZE Foundation for a $10,000,000 prize for a non-government entity to successfully launch a reusable, manned craft into suborbital spaceflight. In the emerging world of data science, data analytics, and artificial intelligence (AI), the Netflix Prize created a competition to design an algorithm to improve predictions about movie preferences. The prize fostered breakthroughs in contest design and analytics design.

Dr. Lakhani described legislative changes and policy interest in the use of prizes. For instance, in 2006, Congress authorized the National Science Foundation (NSF) to establish an inducement prize program.² In 2010 Congress adopted the America Competes Act, which provided prize authority to all federal agencies. Despite an increased interest in and use of prizes, such competitions still remain relatively rare in contrast to the more routine federal funding mechanisms of contracts, grants, and cooperative agreements.

__________________

Thousands of economics papers and textbooks have been written on prize design, and most of the literature is theoretical. The committee’s mandate includes learning from practitioners and adding to the empirical fact base.

Dr. Lakhani outlined the three objectives of the workshop:

• Learn from practitioners about prizes (often referred to as “challenges”) in the federal government and the private sector;
• Develop an understanding of the opportunities and challenges associated with the use of prizes; and
• Build a community of practice.

The workshop included sessions on grants, contracts, and prizes in the innovation ecosystem, international and non-profit experiences with prizes, and lessons learned from implementation of prizes. These were followed by a session on implementing prizes at federal agencies, and a final workshop session focused on the integration of prize outcomes.

SESSION 1: GRANTS, CONTRACTS, AND PRIZES IN THE INNOVATION ECOSYSTEM

Session I was moderated by Alberto Galasso from the University of Toronto, and the panel included Jason Crusan from Woodside Energy, Aniket “Niki” Kittur from Carnegie Mellon University, Philip Brookins from the Max Planck Institute for Research on Collective Goods, and Zorina Khan from Bowdoin College. This panel focused on existing literature around both positive and negative aspects of prize competitions and comparisons of prizes to other policy tools, such as grants or contracts. It included a discussion of the types of prizes that were successful at NASA, highlighting their utility in comparison to other funding mechanisms. Additional discussion included the ability of prize competitions to include more diverse “solvers” when compared to other funding mechanisms—as prize participants don’t necessarily need a specific technical skill or knowledge expertise to contribute to the innovation process, which in turn, helps build a larger community of practice. This was followed by discussion surrounding the disadvantages of prizes, noting the need for more data on the topic to help bridge theoretical and experimental research on prize competitions and their efficacy.

Jason Crusan, Woodside Energy, described his experience utilizing prizes as part of the overall funding ecosystem in NASA. A multitude of tools can be used to acquire knowledge and technical solutions, including contracts (cost-plus or fixed price), grants, and prizes; prizes are not the best tool in every case, he said. In Mr. Crusan’s experience, fixed price contracts work best when NASA can determine specific deliverables and timelines whereas prizes work best when the government can articulate a specific problem requiring solutions which can leverage potential private sector market demand. He stressed the need for an integrated and balanced approach for federal use of contracts, grants, and prizes.

Niki Kittur, Carnegie Mellon University, addressed designing more effective systems for distributed innovation. He cited the Spanish sociologist Manuel Castells’ definition of innovation, which is to “creatively accumulate and recombine knowledge and information in non-routine ways.” For example, a device to assist with the delivery of babies was inspired by a YouTube video on the removal of corks from wine bottles. In a distributed innovation system, multiple people can perform parts of the innovation process (analogical processing).

According to Dr. Kittur, research and development (R&D) experts are typically specialists in one field. Crowd sourcing allows those lacking in specific technical or knowledge expertise to contribute to the innovation process. A contest design that combines R&D experts and crowdsourcing will benefit from both groups’ strengths and may lead to innovation.

Dr. Kittur also highlighted that competitions can be useful to build a community of practice, citing an example when contest losers were invited to serve as advisors on the finalist teams. In that case, 85 percent expressed an interest in serving as peer advisors to the winning teams and over 40 percent actually served on a competitors’ team. Participants found this experience most valuable when their skills and interests were well-matched to a team.

Philip Brookins, Max Plank Institute for Research on Collective Goods, addressed the advantages and disadvantages of prize competition. He noted that prizes, as opposed to other reward systems in a workforce setting, reward relative performance, and spread the effort across a larger group of solvers.

Contest theory indicates that prizes have the following advantages:

• Prizes obtain high-quality solutions to challenging problems;
• Prizes provide a diverse set of solutions to the problem;
• Prizes can significantly hasten the pace of research and development.

Dr. Brookins suggested that to achieve the optimal level of cooperation in solving problems, it might be useful to divide the process into two stages: a competitive stage followed by a less competitive stage.

In Dr. Brookins’ opinion, it would be useful to bridge theoretical and experimental research to more fully understand human behavior in prize competitions. He also noted that the goals for a competition vary, from finding the highest quality solution to a problem to drawing a more diverse set of people to a given field. The objectives of a particular competition affect which prize model is optimal.

Dr. Brookins also highlighted that contestants who don’t win have already invested a substantial amount of time, effort, and/or materials in a competition and posed the question of whether this may lead to reduced participation in contests. He posited that a subsidy or reimbursement to a larger set of participants might be considered to ensure that there are sufficient players in the future marketplace.

Zorina Khan, Bowdoin College, voiced skepticism about prizes. Dr. Khan noted that prizes use administered systems—arrangements where economic decisions about values, rewards, and allocation of resources are made by administrators or panels. Her research indicates that administered systems, as opposed to markets, are generally nonmarket-oriented, lack transparency, and provide no error-correcting process. According to Dr. Khan, the benefits of prize competitions are diffuse, imprecise, and may affect adjacent markets rather than the targeted market. In her view, the most effective incentive for productive change is failure in the marketplace and the most valuable prize is success in the marketplace. She opined that prizes are most effective when used to incentivize activities that are not market-driven.

SESSION II: INTERNATIONAL AND NON-PROFIT EXPERIENCES

Session II was moderated by Diane Burton from Cornell University, and the panel included Cecilia Conrad from John D. and Catherine T. MacArthur Foundation (MacArthur) and Lever for Change, Tris Dyson from NESTA, and Barbara Kowatsch, former Open Innovation Policy Officer in the Directorate General for Research and Innovation under Horizon 2020. This panel focused on what could be learned about prizes from international colleagues and non-profit sector experience. This included a description of the 2016 MacArthur 100 & Change campaign, namely its evaluation and selection criteria, the cost of participating in the competition, and the competition’s outcomes. This was followed by a description of the work of NESTA Challenge, a non-profit based in the United Kingdom that provides expertise to governments seeking solutions that will result in societal benefits. Lastly, the panelists discussed the European Commission’s experience with inducement prizes, which were initiated in 2012 with the goal of attracting new talent. This included details surrounding prize design, award criteria, and assessments of prize success.

Cecilia Conrad, John D. and Catherine T. MacArthur Foundation and Lever for Change, described the 2016 MacArthur 100 & Change campaign to invest $100 million in a project with the potential to address a critical problem of our time. MacArthur engaged external audiences to identify a problem and propose a solution. MacArthur made the evaluation criteria and scoring rubrics publicly available. The four selection criteria were: (1) meaningful (impactful); (2) evidence-based, (3) feasible, and (4) durable. MacArthur engaged a large number of external evaluators, and the final decision was made by its Board. The cost of participating in the competition was the cost of putting forth an idea rather than completing the work, and every applicant received constructive feedback. Further, MacArthur provided the 8 semi-finalists with technical support for scaling and advice.

Sesame Workshop & International Rescue Committee (IRC) won the competition with a joint proposal for early childhood intervention in Syrian refugee areas. Ms. Conrad noted that the competition was the first time that Sesame and IRC had collaborated. MacArthur was so impressed with the other three finalists that it also awarded $15 million to each of them. The competition drew attention to early childhood education, a field that does not typically receive much attention from funders. MacArthur endeavored to bring their finalists’ projects to the attention of other funders and leveraged additional funding, including a LEGO Grant to Sesame and IRC for similar work in Rwanda. MacArthur also shared data and information from the competition as a public resource. Ms. Conrad explained that the second launch of the competition will use external panels at three stages: the first stage will be peer-to-peer review, the second stage will be a cross-disciplinary panel, and the third stage will consist of a review by a panel with technical expertise.

Tris Dyson, Executive Director, NESTA Challenges, described the work of NESTA Challenge, a non-profit based in the United Kingdom that was established six years ago by parent organization NESTA and the British Government. NESTA Challenges provides expertise to governments seeking innovative solutions that will result in societal benefits. NESTA Prize competitions require significant advance thinking about project design and typically consist of several stages, with new entrants at various points of entry. In addition to the prize itself, the challenge provides innovators access to information, ideas, profile-raising opportunities, investment, and expertise. NESTA provides prizes in five areas: global health, better markets (disruptive technologies are steered to maximize benefit), new frontiers, communities, and energy and environment.

NESTA has developed basic principles to determine if a prize is needed:

• Is there a clear goal?
• Is there a benefit to opening the challenge up to new innovators?
• Would a prize definitely motivate innovators?
• Will a prize accelerate progress (rather than just reward past success)?
• Will solutions be commercially viable after the prize is finished?

NESTA uses prizes to create better solutions, bring together innovators, and unlock systematic change. NESTA assesses the impact of its prize by evaluating (1) whether an innovative solution resulted, (2) whether capacities of innovators improved, and (3) whether there has been a broader ecosystem impact.

Ms. Dyson noted that the passage of time is required to fully assess the impact of a prize. On the 300th anniversary of the British longitudinal study, NESTA asked the UK public to vote among several topics for the next competition. The UK public selected the topic of antimicrobial resistance. NESTA is currently funding a competition for development of a diagnostic tool to determine whether one needs to take an antibiotic. Public participation in the voting process raised public awareness of this issue.

Another NESTA competition contributed to the development of a consumer open banking space, and another current project partners with cities, public services, and businesses across the UK to design emergency response plans for the introduction of drones into local communities. In Dyson’s experience, challenge prizes should be conducted in tandem with other activities, such as public awareness campaigns or other funding opportunities.

NESTA has created a variety of challenge designs to achieve different primary objectives and is planning to publish a guide on challenge designs to achieve those objectives.

Barbara Kowatsch, former Open Innovation Policy Officer in the European Commission’s Directorate General for Research and Innovation under Horizon 2020, described the European Commission’s (EC) experience with inducement prizes. In 2011, the EC held a workshop to examine the use of inducement prizes and invited the XPRIZE Foundation to share its experience. The EC decided to initiate prizes to attract new talent (individuals or small companies) not previously involved in EC funding, attract investment in specific areas, spur innovation, reduce the administrative burden on the EC, and raise public awareness. In 2012, as part of the EC’s 7th Framework Programme for Research and Technological Development (FP7), the first prize launched was for alternative ways to formulate, preserve, or transport vaccines. The EC then included inducement prizes in the Horizon 2020 Framework and included prize designs and award criteria in the Horizon 2020 Work Programme. Under Horizon 2020, the following factors are used to determine whether to use inducement prizes to address a particular topic:

• Ease of communication to attract a large number and diverse group of participants;
• Achievability within a reasonable time frame;
• Sufficient scope and depth to allow for many possible solutions and approaches; and
• Sufficiently tailored to allow for verification of results against objective standards.

According to Ms. Kowatsch, the EC has run 24 inducement prizes since 2012. The initial contests were relatively top-down in approach and did not result in as much commercialization as the EC anticipated. The EC changed the approach between 2016 and 2017 and used a more bottom-up approach complementary to Horizon 2020 grants and contracts. The EC is now focusing on awarding a smaller number of prizes at larger amounts to boost breakthrough innovations with market impact. Ms. Kowatsch agreed with other workshop speakers that significant up-front thinking should be dedicated to prize design. The EC has just assessed the impact of its prizes under FP7 and plans to evaluate others. The first prize for a novel technology to produce and transport vaccines awarded to CureVac GmfH, a biotechnology company, is a great success story for the EC. The prize contributed to the company’s receipt of significant additional funding from a foundation and the company’s immense growth in size.

The European Innovation Council (EIC) will be offering other awards to businesses for breakthrough innovations. Among them, it will offer Pathfinder Pilot (grant) and Accelerator Pilot (grant with an option to apply for equity funding) programs to attract small and medium-sized enterprises. Both of these programs will offer coaching and mentoring services.

SESSION III: LESSONS ON IMPLEMENTATION

Session III was moderated by Hila Lifshitz-Assaf from New York University and the panel included Richard Dunn, former general counsel for the Defense Advanced Research Projects Agency (DARPA), Roy Rosin from Penn Medicine, Dyan Finkhousen from GE, and Angelo Rigillo from Enel SPA, Rome. This panel addressed the implementation

and integration of prizes in the context of a larger research and development system. This included a description of the legislative history of DARPA’s prize authority as well as the America COMPETES Reauthorization Act of 2010, which provided all federal agencies authority to award prizes to stimulate innovation with potential to advance the mission of the agency. It was followed by a discussion surrounding the implementation of innovation prizes within corporate organizations, specifically Penn Medicine, GE’s GENIUSLINK, and ENEL.

Richard Dunn described the legislative history of DARPA’s prize authority. Enacted in October 1999, 10 USC Section 2374a authorized DARPA’s use of non-procurement non-assistance agreements to promote advanced research projects. The statutory prize authority was initially specific to DARPA but was subsequently extended to all of the Department of Defense (DOD). Section 845 of Public Law 103-160, as amended, authorized DARPA under 10 USC Section 2371 to fund prototype projects outside of the procurement process.

Mr. Dunn noted in addition to DOD’s,³ NASA’s⁴ and the Department of Energy’s (DOE)⁵ specific statutory authority to conduct inducement prize competitions, the America COMPETES Reauthorization Act of 2010 provided authority to all federal agencies to award prizes to stimulate innovation with potential to advance the mission of the agency.⁶ Agencies should first identify problems that need to be addressed and then survey available legal authorities to determine the best pathway to acquire and apply new knowledge. Mr. Dunn indicated that a team of agency experts is needed to determine which approach to use and how to structure a prize competition.

He also noted that DOD has ample statutory authority to employ inducement prizes as a pathway for further development of technology arising from the prize competition by the prize winner or another prize competitor. According to Mr. Dunn, even in the absence of specific statutory prize authority, agencies could offer prize inducements in conjunction with industry. In his view, the full potential of prizes to attain new knowledge and spur innovation has not been tapped.

Roy Rosin, Penn Medicine, described his experience with innovation prizes within a corporate organization. Penn Medicine has sought innovative solutions within the organization to reduce infection rates and reduce hospital readmission rates. Internal organizational changes require engaging people within the organization. Friction reduction, such as access to legal advice or access to patient records, is a critical part of the innovation process. In his experience, employee passion to resolve the issue and access to resources and time needed to execute vital changes were greater incentives to innovate than prize money. An iterative process for evaluating the impacts of changes is needed, he said. He concluded by stating that inducement prizes have a time and place when there is a clearly defined problem, clear prize criteria, and a need to include external participants in engaging the problem.

Dyan Finkhousen leads open innovation for GE’s GENIUSLINK team. She described inducement prizes within the context of a corporate industrial enterprise. Within the GE context, GENIUSLINK has experimented with open innovation and crowd sourcing to yield increased innovation for GE. GENIUSLINK has accumulated 6 years of experience testing the boundaries of where inducement prizes can be most valuable. It has used digital and prize models to address engineering, material, software, and creative design problems, and challenged assumptions that developing and adopting innovation comes only from within. GE has the option to use its own employees, suppliers, or the open expert innovation market when addressing a challenge. This expert pool, combined with technological solutions, provides GE an elastic innovation operation with a sliding scale between technology and talent. Ms. Finkhousen noted that prize inducement is one of several pathways that can be used to augment innovation. The best pathway will depend upon the goals, operating context, internal assets, and lack of assets to address the challenge. GENIUSLINK has developed a more agile and precise tool for determining which innovation model would yield the best result.

According to Ms. Finkhousen, it is helpful to have a toolkit in place (templates and process and codification) to operationalize a prize. GENIUSLINK borrowed best practices and templates from within the organization and tweaked them for the open competition process. Ms. Finkhousen indicated that these templates enhanced corporate acceptance of inducement challenges.

Angelo Rigillo is the Head of Innovation Governance, Intelligence and Partnerships, ENEL Group, a global renewable energy corporation that provides energy to 70 million people through its networks. ENEL has a vision for open innovation plus sustainability which they call “open innovability.” ENEL has developed a portfolio of open innovation tools, which includes a crowd sourcing solver community, technology communities, innovation hubs, partnerships, and academic partners. (Figure 1)

__________________

According to Mr. Rigillo, inducement prizes are not always the appropriate tool and should be used in conjunction with other innovation tools. ENEL has published challenges and awarded 28 challenge prizes to date in the amount of 350,000 euros. ENEL uses both proprietary platforms for internal ENEL challenges and external platforms to collect ideas from both internal and external actors.

In the opinion of Mr. Rigillo, a successful prize competition requires a clear definition of the challenge so that the problem is clearly communicated to solvers and the judges. It is important to develop internal policies and a standard set of rules addressing issues that arise from inducement prizes, such as intellectual property rights, payments, exclusivity periods, and privacy. It is also important to develop a communication plan, conduct a transparent evaluation process by specifying the evaluation criteria (i.e., technical and economic feasibility, business potential for ENEL, innovation level, and presentation of the proposal), and provide feedback to solvers. Mr. Rigillo recommended that any organization launching a prize use internal judges because they possess the greatest understanding of the need for a solution. Also, the amount of the prize should vary by the complexity of the solution. A larger prize needs to be offered if the competition seeks more than just ideas, such as technical solutions or prototypes.

SESSION IV: IMPLEMENTING PRIZES AT FEDERAL AGENCIES

Session IV was introduced by Zoe Szajnfarber from The George Washington University before speakers divided into breakout groups. Breakout Group A was moderated by Alberto Galasso from the University of Toronto and the speakers included Sandeep Patel from the Department of Health and Human Services (HHS) and Taylor Gilliland from the National Institutes of Health (NIH). Breakout Group B was moderated by Hila Lifshitz-Assaf from New York University and the speakers included Lynn Buquo and Monserrate “Monsi” Roman from NASA as well as Rafael Cotto and Holly Brown from the National Science Foundation. Breakout Group C was moderated by Zoe Szajnfarber, and the speakers included Heather Evans and Sarah Hughes from the National Institute of Standards and Technology (NIST) and Hoyt Battey from the Department of Energy. With the goal of comparing and contrasting the processes being used by a subset of federal agencies, various speakers discussed each federal agency’s use of innovation prizes and provided examples of past or current prize competitions. This included details surrounding the development, approval, and administration of certain challenges, as well as any guidelines or toolkits used by each respective agency—all within the context of a hypothetical topic involving the possible use of a prize/challenge.

Zoe Szajinfarber, The George Washington University, noted that the committee’s goal is to make recommendations on inducement prizes offered by federal agencies. She noted that due to hetereogeneity across agencies in mission, context, constraints, and challenges that the agencies are trying to solve, a common reference problem would offer an opportunity to compare and contrast the processes being used by a group of federal agencies. Prior to the workshop, representatives from each of the agencies sponsoring the study⁷ were given a hypothetical problem an

__________________

agency might face (in this case the existence of a rich dataset that an agency wants to make broadly available but that needs to be cleaned up and made accessible remotely) and were asked to explain for discussion at the workshop the steps that would be involved in implementing a prize to address this topic in their respective agencies.

BREAKOUT GROUP IV-A

Sandeep Patel, Open Innovation Manager at HHS, described the department’s use of innovation prizes to promote health and well-being. HHS has run 90 prizes since 2011 and awarded $41 million in cash prizes. HHS is currently running 20 to 30 prizes per year and has increased the prize amount over time to incentivize sufficient participation in the competitions. HHS uses a broad range of prize competitions to encourage rapid development of novel ideas, reach new innovators, and reduce time or cost. Prizes can be used to develop specific project components (such as software), advance general science and technology development, create business plans and start-ups, and perform science data and analytics in new ways. (Figure 2) Recent examples of HHS prizes include new technologies for remote pregnancy monitoring, development of tools to connect health systems, and advances in the state of art of kidney disease products.

Following the passage of the America Competes Act in 2011, HHS developed a policy for prizes and hired an open innovation manager to implement the use of prizes at HHS. HHS created a playbook resource for designing a prize and measuring the ultimate impact.⁸ HHS ran challenge boot camps in 2015 and 2016 for HHS employees, and interested employees have formed a community of practice. Further, HHS’s Innovation Office provides advice on how to develop a challenge while individual agencies within HHS implement specific challenges.

In its 2011 policy, NIH clarified that prizes can only be awarded to U.S. entities and determined that judges should include HHS employees and outside experts. The policy also differentiates between prizes and other funding mechanisms and clarifies when HHS funds are obligated for a prize. Under the policy, HHS assigns roles for HHS offices within the prize process, including selection of an HHS employee to serve in the role of challenge manager and as an award-approving official at the beginning and end of the process.

Taylor Gilliland, NIH Challenge Manager, outlined NIH’s use of prize challenges. NIH has 27 institutes and centers, 24 of which receive their own appropriations from Congress. Eighty-five percent of NIH’s budget is awarded as grants, contracts, and cooperative agreements. NIH uses prizes to establish an ambitious goal within a relatively short time frame, paying only for results and without bearing high levels of risk. According to Dr. Gilliland, prizes have been used to draw public attention to a public health need, increase the number and diversity of solvers, encourage application of NIH research data to address public health needs, or stimulate private sector investment in a critical need area. Ongoing NIH challenges include a $20 million initiative to develop a diagnostic test to detect antibiotic-resistant bacteria, a design challenge for translational innovations to address the opioid epidemic, a National Institute of Aging

__________________

prize for a software application to improve coordination among caregivers for Alzheimer patients, and a high school essay contest regarding students’ experiences with mental health. To date, NIH has run more than 45 challenges and awarded $13 million in cash prizes.

Responding to the data challenge problem, Dr. Gilliland stated that the first step would be to consult NIH’s updated interim policy for prizes. The interim policy establishes procedures for developing, reviewing, approving, and administering NIH-sponsored challenges, including eligibility requirements, intellectual property allocation, judging criteria, and conflicts of interest. He noted that NIH’s prize authority is derived from the America Competes Act of 2010, as modified by the American Innovation and Competitiveness Act (AICA) of 2017.⁹ All challenges must be posted on Challenge.gov. The challenge manager, delegated responsibility for managing prizes by the respective institute director, may engage the program director, budget officer, policy officer, communications team, and IT team in forming a challenge team. As a challenge team, they decide whether to use a cash or non-cash prize, whether the competition should be run internally or by an external entity, who should judge the competition, and what will happen post-award. (Figure 3) Who has authority to approve a challenge depends of the size of the prize. (Figure 4)

__________________

BREAKOUT GROUP IV-B

Lynn Buquo, NASA’s Center for Excellence for Collaborative Innovation (CoECI), explained that NASA has two different models for prize competitions. One utilizes NASA’s procurement authority while the other is based upon NASA’s prize authority. CoECI uses procurement as its tool for crowdsourcing and provides advice to NASA challenge owners on the appropriateness of using an open competition to solve a particular challenge. The challenge owner is responsible for funding the competition.

NASA has a cooperative agreement with Harvard’s LISH for data science support for its prize competitions. Working with the challenge owner and LISH, NASA develops a Request for Information to develop a cost estimate and validate the challenge scope. A Request for Proposals is then released, a company is selected, and a fixed price task order completed. A NASA challenge coordinator is assigned to serve as project manager.

NASA conducts a one-year follow-up after the competition to evaluate the outcome. NASA has run over 300 challenges to date, and over 90 percent have resulted in obtaining useable solutions. In Dr. Buquo’s experience, open challenges are used more frequently by NASA engineers than scientists. NASA also assists other federal agencies interested in conducting inducement prizes.

In contrast to CoCEI, Monsi Roman, Program Manager, NASA’s Centennial Challenge Program (CCP), explained that the CCP is run under NASA’s statutory prize authority. CCP uses challenges to stimulate innovation in research, technology development, and prototype demonstrations. NASA designs the challenge, and the competitors retain the intellectual property rights. NASA currently has five centennial challenges: tissue reengineering, communication and propulsion, robotics, mining resources, and conversion of carbon dioxide into glucose. NASA has used prize competitions in the past to design 3-D habitats on the moon and 3D printable objects for astronaut’s medical needs in space.

Venture capitalists can also participate as partners in the competitions. These prize competitions solve high-technology gaps and create a community of experts. According to Ms. Roman, it is important to carefully craft the competition design and clearly communicate the challenge to solvers; evaluation criteria are clearly defined before the competition. NASA cannot award funds to international competitors, but international partners can participate in these open challenges. Some of the winning technologies can be used in non-space applications until NASA is ready to pick up these technologies for deployment in space.

Rafael Cotto and Holly Brown, National Science Foundation, explained that the agency relies upon the NSF Organic Act of 1950 (42 USC 1861 et seq.) and Title IV of the American Innovation and Competitiveness Act for its authority to conduct prize competitions. NSF is exploring how to use prize competition methods as an option, in addition to contracts, grants, and cooperative agreements, to advance NSF’s mission. In making this determination, NSF examines the purpose of the competition, what other funding mechanisms are available, whether a prize will engage a new audience not already part of the NSF community, and whether there are any sensitivities in using a prize competition. NSF is currently developing internal guidelines and toolkits to guide staff whether to use a prize competition mechanism. With respect to the reference problem, they said that data cleaning may be handled internally by NSF experts, or it might be handled through a contract.

NSF has historically used prizes to engage a broader audience. A past NSF challenge competition focused on community colleges and required students, an instructor, and industry partners to solve a problem within their community. NSF is currently running the 2026 challenge for ideas on what is the next big area of science. Some of the interesting submissions have been from individuals in fields that are science-adjacent such as artists who do medical illustrations or a sculptor that created life-like models for forensic analysis. NSF noted that it is not required to choose a winner if one does not emerge from the competition.

BREAKOUT GROUP IV-C

Addressing inducement prizes at NIST, Heather Evans noted that NIST conducted its first cash prize competitions in 2015 and has steadily increased its use of prizes. NIST has a large and diverse intramural research enterprise. To date, NIST has run a total of 15 challenges and awarded $5.5 million in prizes. Prizes have been used in the areas of health, public safety, and manufacturing. As the NIST Coordinator for Prize Competitions, Dr. Evans partners with a staff researcher and tries to act as friction remover in the prize process. In the design phase, experts familiar with the challenge are consulted. Workshop mechanisms also can be used to assist NIST with gathering community input about the issue and how to solve it. Evaluation criteria are developed during prize design and reviewed by NIST’s Office of General Counsel. The evaluation criteria are posted publicly at the time the prize is launched on Challenge.gov. NIST uses webinars and marketing pushes to promote awareness of its competitions.

Sarah Hughes, of NIST’s Public Safety Communications Research (PSCR), described the use of prizes by PSCR to drive public safety communication technologies for first responders. PSCR works in partnership with First Responder Network Authority to assist in the development of technologies for first responders and also serves as an early-stage research and development entity. The life cycle for NIST’s PSCR Prize Challenges consist of ten stages (Figure 5), and the prize approval process goes through different levels of approval, with final authority given to the Director on prizes over $1 million.

With respect to post-award support, PSCR has two pilots for its FY 2019 prizes to offer technical assistance to awardees. In Ms. Hughes’s experience, the federal research laboratories at NIST have remarkable talent and resources, and prizes provide opportunities to make these resources available to partners from innovative entities.

Responding to the data reference problem, Ms. Hughes noted that NIST has authority to provide standard reference data and maintain data repositories for its community. NIST has in-house expertise in data science and might choose to engage in the data infrastructure development itself to enable it to host a long-term resource for the NIST community. Alternatively, NIST could contract a vendor to transform data into a shareable format or use a prize if it seeks outside thinking to address the problem.

Hoyt Battey, Department of Energy (DOE), explained that although DOE has been running prize competitions since 2015, they constitute a small percentage of Department funding. His office on Water Power Technologies ran a Wave Energy Prize in 2015-2016 (which focused on catalyzing the development of wave energy converters with the aim of reducing the cost of wave energy) and is launching a new Waves to Water multi-stage competition focused on wave power technologies. There is no defined review process for prizes at DOE, but Mr. Battey estimates that it would require approximately seven or eight signatures. In the Water Power Technologies’ office, approval to run a prize has been delegated to the Assistant Secretary for prizes below $1 million and the Undersecretary for prizes above $1 million. DOE needs to understand potential current and future uses to determine whether it is appropriate to use a prize competition to solve the problem. An ideal competition might be held at an initial stage of a problem to gain a greater understanding all the potential uses and platforms.

DOE employees interested in prize competitions have formed an informal community. An informal best practice at DOE is to announce preliminary plans to run a competition and work with public stakeholders to design the prize competition before the official launch. At the early stages of a prize (ideation), more subjective evaluation criteria are needed while more objective criteria can be used at later stages (prototypes). Prizes are well-suited when a desired outcome is cooperation and sharing of information among contest competitors. Mr. Battey stated that DOE has found that when they choose to use a prize, outcomes have been better than what they would have achieved through a traditional grant.

SESSION V: INTEGRATING PRIZE OUTCOMES

Session V was moderated by Anne-Laure Fayard from New York University and included opening presentations by James Beck, a Wendy Schmidt Ocean Health X Challenge winner, David Hertz, a Water Abundance X-Prize winner, and Marybeth Haneline from Nova Labs, Inc. After opening presentations, speakers divided into breakout groups. Breakout Group A was moderated by Jack Hughes from Lewis Bay Holdings, and the speakers included James Beck and David Hertz. Breakout Group B was moderated by Anne-Laure Fayard, and the speakers included Marybeth Haneline, Daryl Peace, and Patrick Thompson, all from Nova Labs, Inc. While the first breakout group focused on two winners of XPRIZEs, specifically focusing on their experiences throughout the prize competition, the second group focused on open competitions conducted by a Washington DC-area “makerspace” (Nova Labs, Inc.) that bring together diverse sets of individuals to develop assistive technologies. This group discussed the ways in which the non-profit assists with open collaboration for “makers.”

BREAKOUT GROUP V-A

James Beck, Sunburst Sensors LLC, discussed his team’s work on measuring carbonate systems in oceans. Sunburst won the Wendy Schmidt Ocean Health XPRIZE for instruments that measured carbon dioxide/pH at various depths. Sunburst Sensors’ team won both the accuracy and affordability purses. Competitors came from eight countries and included engineers, oceanographers, and professionals in the food industry and agriculture. The competition provided Sunburst an opportunity to test their instruments, cooperate with other teams, and work with other foundations to improve measurement of ocean acidity.

David Hertz led the winning team of the Water Abundance XPRIZE, which was designed to enhance access to potable water in locations lacking infrastructure following disasters. The $1.75 million prize required contestants to produce 2,000 liters of potable water from air with renewable energy at a cost of under 2 cents per liter. Mr. Hertz organized a team of collaborators that used biomass gasification as the energy source. The XPRIZE Foundation is increasingly focusing on implementation of its prizes and provided the team with funds for consulting fees for its strategic plan. The team is exploring microfinancing options to enable communities to have a local sense of ownership of these utilities. With respect to intangible assets, Mr. Hertz noted that the XPRIZE increases public awareness of technologies, builds the winner’s confidence, enhances the team’s credibility, and provides new opportunities for strategic alliances and collaborations. The competition drives a competitive spirit, and the deadline hastens innovation. In his view, the prize is a force multiplier that goes beyond the cost of administering the prize. Mr. Hertz noted that “The prize is not the goal. It is a process, and not an event,” and is a lever for positive change in addressing global challenges.

BREAKOUT GROUP V-B

Marybeth Haneline, Daryl Peace, and Patrick Thompson. Nova Labs, Inc., is a non-profit, member-supported makerspace: run by volunteers, it provides space, tools, and machinery for member “makers” in an environment of open collaboration. In addition to providing a creative space for members, Nova Labs has engaged with the broader surrounding community in a variety of ways, including skills development programs with schools and local businesses and “make-a-thons,” multi-day events in which teams of individuals with a diversity of skills and interests come together to develop innovative solutions to discrete challenges. Participants in “Make-A-Thons for Assistive Technology” compete in teams to build working prototypes of solutions to specific physical challenges. In one example, make-a-thon participants developed a new prototype for a walker with a bucket seat and retracting belt that enabled an individual to pivot and shake hands. In others, participants used sensors to enable a blind individual to ride a skateboard and developed a custom seat for a golf cart to enable an individual who cannot stand up to play golf. The participants and sponsors to date have chosen to open-source the results arising from these competitions.

CLOSING REMARKS

In closing the workshop, Dr. Lakhani noted that prizes can be an effective means to solve complex issues and stimulate innovation. The workshop provided insight into the historical use of prizes and the evolution of prizes as a tool for innovation. The full National Academies study will provide an opportunity to examine the role of prizes within the portfolio of tools available to government and industry to induce innovation.

WORKSHOP PLANNING COMMITTEE: The following members of the Committee on the Role of Inducement Prizes planned the workshop: Karim Lakhani, Chair, Harvard Business School, Alph Bingham, M. Diane Burton, Cornell University, Anne-Laure Fayard, New York University, Alberto Galasso, University of Toronto, Jack Hughes, Lewis Bay Holdings, LLC, Hila Lifshitz-Assaf, New York University, Stephanie S. Shipp, University of Virginia, Zoe Szajnfarber, The George Washington University, Norman Whitaker, Microsoft Corporation, and Brian Wright, University of California, Berkeley.

DISCLAIMER: This Proceedings of a Workshop-In Brief was prepared by Anita Eisenstadt, Meghan Ange-Stark, and Gail Cohen as a factual summary of what occurred at the workshop. The statements made are those of the individual workshop participants and do not necessarily represent the views of all workshop participants, the planning committee, the Laboratory for Innovation Science at Harvard, or the National Academies of Science, Engineering, and Medicine.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Cristin Dorgelo, Association of Science-Technology Centers; Georgina Campbell Flatter, Climacell; and Neil Thompson, Massachusetts Institute of Technology. Marilyn Baker, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

SPONSORS: This symposium was sponsored by the Department of Energy, Department of Homeland Security, National Institute of Standards and Technology, National Institutes of Health, National Aeronautics and Space Administration, Department of Health and Human Services, and National Science Foundation. The National Academies thanks the Laboratory for Innovation Science at Harvard for its cooperation in organizing the workshop.

For additional information regarding the workshop, visit http://nationalacademies.org/step.

Suggested citation: National Academies of Science, 2020. The Role of Inducement Prizes: Proceedings of a Workshop – In Brief. Washington DC: The National Academies Press. https://doi.org/10.17226/25892.

Policy and Global Affairs

Copyright 2020 by the National Academy of Sciences. All rights reserved.