National Academies Press: OpenBook

Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop (2017)

Chapter: 8 Key Themes and Possible Next Steps

« Previous: 7 Innovation Measurement Agendas of the Future
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

8

Key Themes and Possible Next Steps

Scott Stern and Bronwyn Hall closed the workshop with comments on what they saw as some of the main points presented and discussed. Stern noted three broad questions were addressed, in some cases directly and in others indirectly:

  1. What can be done to facilitate productive updating of the Oslo Manual, which defines innovation and prescribes an approach to its measurement, going forward?
  2. How should metrics be developed and modernized to capture the components of innovation—including its inputs, outputs, and dynamics over time—more accurately and more comprehensively?
  3. How are new (or revised) measures and datasets changing the understanding of innovation?

Stern noted that an objective of the workshop organizing committee was to envision how the range of measures could potentially be improved and expanded to more fully capture, on a reliable basis, aspects of innovation processes that are currently missed or not completely measured. This goal requires first taking stock of the work already being done by the National Center for Science and Engineering Statistics (NCSES) and other statistical agencies. The value of their contributions, made possible by significant public investments, improve the understanding of innovation and advance its measurement and was clearly evident at many steps

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

along the way during the proceedings. While highlighting the value for new kinds of data and new analytic constructs, Stern credited workshop participants for recognizing that the current apparatus—for example, the Community Innovation Survey (CIS) and the Business R&D and Innovation Survey (BRDIS)—contains invaluable tools that benefit from a balance between standardization (across countries and time) and experimentation (to uncover new phenomena and local circumstance). Stern made the point that planners should be explicit about what parts of a survey and resultant data products require standardization (because the capacity to make direct comparisons across countries and time has value) and what parts could be opened up to experimentation.

While progress in the field was acknowledged, several workshop participants sought to identify what Ben Martin (Chapter 2) termed the “dark matter”—aspects of innovation that are not measured and, consequently, not well understood. Over the course of the presentations, Stern observed, it became clear that shedding light on this dark matter will entail individual and organizational initiatives on domains outside the current scope of the Oslo Manual and the CIS upon which it is based. In some cases, the comparative advantage for advancing measurement will be in collaboration with the statistical agencies, while in other cases, researchers will need to take the lead.

Regarding revision of the Oslo Manual, Stern pointed out that the definition of innovation matters. Fred Gault (Chapter 2) commented that the definition is central because it is the anchor determining what is in scope and what is out of scope. Thus, to some extent, a narrow definition will confine inquiry and a broad definition will stimulate wider inquiry.

A major theme that emerged from this workshop—reiterated in the presentation by Hall—was that, to be useful, any framework must be capable of reflecting the complex and diverse processes and interactions inherent in innovation:

Science, technology, and innovation consist of a number of components linked by the knowledge and resources that flow among them. These components include governments, government research laboratories and extension services, the intellectual property system, higher education and research institutions, venture capital, and industrial research laboratories. Perhaps less obviously, they also include individuals across many economic sectors who are engaged in improving the efficiency of production, introducing new ideas and new products, or attempting to adopt new technologies or methods of organization. When the system works well, the interaction of these institutions and individuals [turning ideas into value] produces welfare-enhancing economic growth via the introduction of new and improved processes, products, and services (Hall and Jaffe, 2012, p. 34).

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

Ideally, Hall argued, a measurement framework would specify a system of indicators that, taken together, are helpful in monitoring these processes and informing policies designed to foster, facilitate, and enhance them. The conceptual framework drives (implicitly or explicitly) data collection and data organization, as well as what measurements based on those data will be illuminating. Subsequently, when measurements are used in research, a feedback mechanism is created that further shapes the framework and helps to identify data and indicator gaps that, if filled, could help answer key questions about the pace, location, and nature of innovation activity (Hall and Jaffe, 2012).

Though the idea of a single unified indicator framework has allure, it is not possible if the goal is to promote research on the many ways that innovation is fostered and impacts the economy and society. This point was raised in the presentations during the workshop session on “regional innovation models and data needs.” At subnational levels of analysis, it becomes especially clear that multiple approaches are needed to serve the many kinds of users of innovation (and science and technology) indicators and to address the wide range of policy questions posed. No one set or system of indicators will suit all monitoring, benchmarking, evaluating, and forecasting purposes—a unified framework is probably not a realistic goal. The range of objectives across stakeholders represented at the workshop dictates that multiple approaches may be considered when designing data collection and indicator development programs. For example, as noted above, since some potential benefits of innovation—such as better health, greater happiness, and improved environment—are not market based, other concepts of output (and sets of indicators to capture their trends) are also needed.

Accounts of innovation are also increasingly capturing a broader range of sources and administrative units. As discussed by Owen-Smith, von Hippel, and Winston Smith, among others (see Chapters 5 and 6), innovation can be tracked at the industry and sectoral levels but, in principle, it can also be traced back to where it originates, which may be businesses, households, individual entrepreneurs, universities, nonprofit organizations, or the public sector (or some combination of these). The Oslo Manual was created largely for the production of national-level and industry-level innovation statistics, which are important. Yet several presenters (see Chapter 6) demonstrated that measurement can be applied with illuminating effect at more localized levels—levels that are often more relevant in terms of understanding the sources of innovation (such as the human capital and geographic clustering of activities) and their impact on economic and social outcomes.

Additionally, measurement frameworks should reflect linkages in innovation processes so that flows of inputs, outcomes, and broader eco-

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

nomic and societal impacts can be tracked. Charles Edquist (Chapter 5) noted the importance of distinguishing innovations from their determinants. He viewed entrepreneurship and research, for example, as drivers of innovation. He said these determinants need to be measured, as do the innovations themselves and the impact they have, in a detailed and coordinated way.

Regarding modernizing innovation measurement and metrics, Edquist reiterated that the scope of data collection must continue to broaden beyond R&D and patents. Stern noted the modal notion is that R&D is undertaken in a company, which then leverages the knowledge generated to produce one or more innovations—suggesting an alignment between R&D as the input with the output being new or significantly changed goods and services produced by the company. This perception may have been sufficiently accurate at the time when a number of key surveys were designed, but the interaction of different players is now more complex. To portray innovation fully, data systems should be capable of detecting knowledge linkages between people and companies—and business and household interactions generally—over time (as noted by Javier Miranda, Chapter 3), and the division of labor, which means that people engaged in idea development are often different than those commercializing the innovations (Wesley Cohen and Ashish Arora, Chapter 3). In addition, Stern concluded that the role of “nontraditional” but prominent aspects of value creation, such as design (Bruce Tether, Chapter 3), digitization, and robotics (Rob Seamans, Chapter 3) is necessary. These complexities are only compounded by the changing innovation landscape that includes ever-increasing internationalization, globalization, networking, and co-invention. Similarly, Robbins (2016, p. 11) in the background paper prepared for the workshop noted, “the increasingly open nature of innovation, including the role of suppliers, customers, licensed technology, and collaborations and joint projects, suggest that the activities of firms need to be understood in the context of their relationships to other institutions, including other firms, universities, non-profits, and government entities.”

In her summary comments, Hall noted that a comprehensive portrayal requires acknowledging that innovators (and not just the innovations they produce) matter: that is, a recognition that knowledge flows via individuals. Innovation is undertaken by a person or teams of persons, which makes maintaining individual identifiers in datasets crucial to tracking these linkages and for anticipating where action may occur. This point was made by Eric von Hippel (Chapter 6), Sallie Keller (Chapter 7), and others: People are innovators. Scientists, engineers, and designers, as well as people in the broader community, can be identified as engaging in a distinctive human trait, which is making tools, using tools, problem solving, and developing novel creations. Winston Smith commented that

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

this notion of tracking innovators as well as innovations is important at both the level of measurement (e.g., suggesting a need for data on the migration of science, technology, engineering, and mathematics graduates to firms) and at the level of the questions asked.

Ben Martin (Chapter 2) reinforced discussion to move beyond traditional indicators. In particular, he argued that indicators based on patent data are often only distant proxies for the true measurement objective. Jeff Oldham (Chapter 7) also demonstrated that patent data sources could be used more effectively, by linking them to nonpatent data, to shed light on the impact of patented ideas on sales, quality of life, educational effectiveness, and other areas. Several participants argued it was time to broaden sectoral coverage of innovation. Services and experiences increasingly dominate value-added and employment in the United States and other advanced economies, they pointed out, but there is still a manufacturing orientation in many measures of innovation. Martin noted, for example, that if more had been known about the characteristics of innovation in the financial sector, policy makers and researchers would have been better placed to anticipate the events leading to the 2007 crisis. This also highlights that not all innovation is socially desirable.

Several presentations highlighted that comprehensive conceptualization and measurement of innovation also requires recognizing activities (and outcomes) that take place in households (Eric von Hippel, Chapter 5), in the public and nonprofit sectors (Kay Husbands Fealing, Chapter 5), and in communities (all presenters in Chapter 6). Stern remarked that a goal for the future should be to better capture—in both survey-based and administrative datasets as well as the broader programs of the National Center for Science and Engineering Statistics, OECD, and others—the scope of innovation in a way that integrates leading-edge research findings (which breaks new ground) into the statistical machinery (which facilitates consensus and common understanding).

Including the public sector in “official” innovation measurement raises an important issue (which also applies to some other sectors), which is the dearth of data on outcomes. Hall pointed out that measurement of outcomes for public-sector innovation requires thinking about how survey questions should be asked. The wording of survey instruments has to change not only for the public sector and not only for the innovation question, but also possibly for other questions, she noted. Measurement of the innovation and public goods link (which may take place in the public or private sector) requires connecting indicators of the inputs (mainly dollars and human capital) all the way through to social outcomes in the ecosystems in which they take place. As von Hippel noted (Chapter 5), much of the current focus is on economic outcomes or tangible outputs, which leads to an overemphasis on patents and other readily available

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

data; many of the useful innovations that bring high utility for the public good are not “patented ideas.”

The development of innovation measurement will, to a great degree, be driven by the availability of new kinds of data—where, as Stern phrased it, the administrative data revolution meets the digitization revolution. The administrative data revolution discussed by Javier Miranda (Chapter 3) reflects at least two developments. One is an increasing capacity at the statistical agencies and others to link different administrative and survey data sources; Stern cited the work by Chetty et al. (2014) combining IRS earnings records and other sources to examine determinants of intergenerational economic mobility as one high-profile example. Elsewhere, there are data on entrepreneurship, around health records, around studies of schools and teacher value added, and others. In addition, Stern observed, the digitization revolution allows researchers to uncover new ways to understand company activities, new products, and new services that reflect the process of innovation. Except for a few examples, these two developments have so far existed as separate tracks within the innovation community. Sallie Keller (Chapter 7) provided an overview of the role of nonsurvey data for statistical use, and of how statistical and data-gathering techniques are already being used to advance research interests.

As articulated by Robbins (2016), innovation-related data priorities for the statistical agencies can be informed by evaluating the needs of the data-using community against three factors: (1) data currently available from NCSES and other statistical agencies, (2) increasing opportunities provided by nonsurvey data, and (3) opportunities for linking existing datasets. Further, capabilities of the data-collecting agencies can be enhanced for these priorities by leveraging partnerships, collaborations, and supporting research. Several participants identified enhancing the ability to link data across sources as a prerequisite to analytic and measurement progress. One example among the many kinds of new data that will aid a greater understanding of innovation are those tracking angel funding and crowd funding, but currently, the data are still not sufficiently detailed and available to enable sustained high-quality research on new innovation funding.

Regarding what has been learned and continuing gaps, Stern observed that researchers are able to measure some aspects of innovation extremely well. However, he said, the share of what is being missed—the portion identified by Martin’s dark matter analogy—is still largely unknown, except perhaps in a highly aggregated sense as reflected in statistics on productivity (as described by Dan Sichel, Chapter 3). And productivity measurement is essentially a black box—a residual measure of what is not explained by changes in the levels of inputs of production. Even assessing the amount missed on the basis of productivity measures (without claim-

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

ing to identify the sources) is controversial, Stern noted, as it requires anchoring one’s belief that the input-output relationships are captured accurately, which is not universally accepted.

Stern described the following areas for future research identified by workshop participants as key to shedding light on some of the innovation “dark matter”:

  • The division of innovative labor: Research by Cohen, Arora, and others is demonstrating that a surprisingly high fraction of important innovations come from outside the firm, albeit sometimes cooperatively, Stern said. The linkages among innovating individuals and organizations need to be better understood.
  • Design, digitization, robotics, household users and entrepreneurship: Workshop discussants (specifically, Tether, Chapter 3) noted that aspects of innovation such as design could be systematized enough to be measured at the level of a “design index.” Rob Seamans’ presentation (Chapter 3) on innovations in the area of automation illustrated that there is not a single use dataset that currently captures job-creating and job-replacing phenomena adequately.
  • Centrality of the regional systems approach: A considerable amount of new work is seeking to understand entrepreneurship and other innovative activities at the regional level. Practitioners and policy makers have become accustomed to the idea that innovation statistics are produced at the national level. In the 1990s, outside of a few state-level aggregates, science and engineering indicators were based at the level of countries. Stern observed that at least in a large country such as the United States, many processes can be better understood at the regional level. Presentations by Feldman, Fazio, Ziedonis, and Winston Smith (Chapter 6) made it clear that there is probably more to be learned at the subnational unit of analysis than at the national level, especially because both research communities and policy communities are often regional in character.

The last area that Stern noted as emerging from the workshop concerned the skewed distributions of discovery, invention, innovation, and entrepreneurship and the implications for data and research. Stern referred to an empirical phenomenon that a large percentage of profoundly influential innovative activity—for example, returns on initial public offerings or the impact of ideas on profits, products, and job creation—occurs from a small proportion of the distribution. This means that, for some measurement purposes, detailed statistics based on 1 percent of activity (if it is

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

the right 1%, or perhaps 2 or 3%) are very useful, while information on the full distribution of total number of papers or patents or research and development that a country or a region produced is inefficient in producing insights. At an aggregate level, he said, it may be more important to know if a country or a region is reliably producing innovations that appear in the top 0.5 percent of the distribution measured along dimension x than to know their totals. An important research priority for the near-term horizon, then, is to evaluate what statistics can be produced at the nth percentile as part of the information infrastructure. The Census Bureau’s Business Dynamics Statistics group has done pioneering work to reveal the analogous skewedness of job creation and destruction, firm survival, and other areas. Stern suggested that this work might provide a blueprint for further work on the topic. Just as focusing on the top 1 percent in the area of income changed the conversation of what people studied, focusing more centrally on these skewed distributions in areas related to innovation is important for advancing understanding of the relationship between innovation and its sources and outcomes.

Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 99
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 100
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 101
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 102
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 103
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 104
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 105
Suggested Citation:"8 Key Themes and Possible Next Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 106
Next: References »
Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop Get This Book
×
Buy Paperback | $64.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Because of the role of innovation as a driver of economic productivity and growth and as a mechanism for improving people's well-being in other ways, understanding the nature,determinants, and impacts of innovation has become increasingly important to policy makers.

To be effective, investment in innovation requires this understanding, which, in turn, requires measurement of the underlying inputs and subsequent outcomes of innovation processes. In May 2016, at the request of the National Center for Science and Engineering Statistics of the National Science Foundation, the Committee on National Statistics of the National Academies of Sciences, Engineering, and Medicine convened a workshop - bringing together academic researchers, private and public sector experts, and representatives from public policy agencies - to develop strategies for broadening and modernizing innovation information systems.This publication summarizes the presentation and discussion of the event.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!