National Academies Press: OpenBook
« Previous: References
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

Appendix A

Workshop Agenda

NCSES/CNSTAT Workshop on Advancing Concepts and Models of Innovative Activity and STI Indicator Systems

National Academy of Sciences
2101 Constitution Ave. NW, Washington, DC 20418
Conference Room 120

Thursday-Friday, May 19-20, 2016

Workshop Goals

The workshop will bring together academic researchers, private- and public-sector experts, and public policy agencies to develop strategies for broadening and modernizing science, technology, and innovation (STI) information systems. The workshop will be oriented toward helping the NSF’s National Center for Science and Engineering Statistics (NCSES) refine and prioritize its work on innovation metrics to maximize the relevance and utility of its data collection programs and statistical products to users. The focus will be on conceptualizing innovation—its inputs, outputs and consequences—in a way that reveals elements that are being measured well and those that are being measured inaccurately or not at all. Presentations and discussion will take into account the role of innovation, not only as it affects economic growth and productivity directly, but also as a mechanism for creating greater public good and meeting social challenges such as those associated economic mobility, health, civic engagement, population aging, or climate change. Workshop participants should also imagine how new kinds of information—e.g., naturally occurring, unstructured, digital data—may be used to complement more traditional survey and administrative data in the construction of innovation metrics. Participants should seek to identify questions that cannot be answered now but could be with additional data that have a reasonable chance of being collected.

Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

Thursday, May 19 (9 am-5 pm)

9:00 am Welcome; Project Objectives
(breakfast will be available outside conference room 120)
Key issues pertaining to the changing face of innovation and potential approaches to its measurement will be identified. NCSES will provide an overview of the agency’s goals and future directions, recent work, and the variety of studies initiated on innovation and human capital data. The focus on concepts and models in this workshop is meant to generate input to data development strategies that take into account both user needs and emerging opportunities.
  • – Scott Stern (MIT; workshop chair): Key questions, goals for the day
  • – John Gawalt (NCSES): The scope of innovation measurement in the context of the agency’s user community—e.g., for international comparisons and benchmarking, policy, academic research, business uses. Reflections on the NAS Capturing Change report
  • – Brian Harris-Kojetin (Deputy Director, CNSTAT): Welcome; overview of the Academies
9:15 Assessing Innovation Measurement: How Accurately Do We Measure Innovation Processes and the Resultant Outcomes Delivered to Society and the Economy?
Traditional measures of innovation, based largely on data reflecting expenditures on inputs (e.g., investment in R&D), capture the impact of innovation on economic outputs and outcomes in only a partial way. This has led for calls to collect new kinds of data and create new metrics. In this session, participants will identify aspects of the process and its impacts that are currently being measured well as well as those that are being measured incorrectly, incompletely, or not at all. Such assessment encompasses inputs to innovation (R&D and other), outputs and outcomes of innovation (economic growth, better functioning society), and the utility of metrics to stakeholders ranging from government to businesses. The takeaway from this discussion should be guidance to NSF about the extent to which the right data are being collected, and what additional kinds of information should be pursued. [15 minutes each]
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
  • – Bronwyn Hall (UC Berkeley; session lead): The role and purpose of indicator frameworks in the analysis of science, technology, and innovation and economic well-being; potential impacts and relevance of extending frameworks to include public-sector innovation and broader social impacts
  • – Ben Martin: (University of Sussex): How emergence of the knowledge society shapes innovation; implications for measurement of innovation
  • – Fred Gault (UNU-MERIT, Maastricht): Developing new indicators to capture the changing nature of innovation; policy and other uses of innovation data and indicators; international comparisons
  • – Fernando Galindo-Rueda (OECD): International standards for measuring and comparing innovation; public procurement of innovation; status and overview of Oslo Manual revision
  • – Dan Sichel (Wellesley College): Innovation and productivity—recent puzzles
Open Discussion [15 minutes]
10:45 Break (refreshments available outside conference room 120)
11:00 Innovation Beyond R&D
Data on narrow aspects of innovation—such as R&D expenditures or patent applications, assignments, and citations—are often used to proxy levels of innovation activity. However, in the measurement community, it has long been understood that activities both technical and nontechnical, including those that are not a directly measurable function of R&D and science, also play an important role in innovation.
NCSES’s Census Business R&D and Innovation Survey (BRDIS) asks firms for information about newly introduced or improved products and processes, and about purchased R&D inflows and outflows, revenue from the sale of patents and patent licensing, and a number of IP-related activities. Some questions about innovative activities can be adequately addressed with current data sources such as BRDIS, while others may require new or different kinds of data. For example, less is known about how innovation takes place in organizations that are not R&D intensive, as is often the case in the service sectors. Questions to be addressed here include: What models exist for assessing different kinds of (e.g., non-
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
R&D-based) innovation? What do we know about innovators acquiring inventions from external sources? How would understanding the division of innovative labor affect our understanding of the propensity to innovate? [15 minutes each]
  • – Wesley Cohen (Duke University; session lead): Recent developments in survey-based approaches to measuring innovation, including the Community Innovation Survey
  • – Ashish Arora (Duke University): Technology licensing as an alternative to in-house R&D
  • – Javier Miranda (U.S. Census Bureau): Research using LBD/BDS and other statistical agency data that helps measure innovation.
  • – Kristin McCue (U.S. Census Bureau): New and emerging data products at statistical agencies related to innovation [Ask for documentation of current data products along with a summary of things the agencies are planning to roll out (or considering) for the future]
  • – Bruce Tether (University of Manchester): Design and development—design as an alternative or complement to R&D
  • – Rob Seamans (Council of Economic Advisers): Innovation, technology, and industrial organization; tracking automation across sectors of the economy to inform employment and other policies
Open Discussion [15 minutes]
12:30 pm Lunch (available outside conference room 120)
1:30 The Role of Individuals (and networks of individuals) in Innovation
Compared with institutions and organizations, less attention has been given to the role of individuals and teams in the innovative process. How people’s education, entrepreneurial talents, and other human capital characteristics result in innovation are vast and largely unresolved research areas. One example of how innovation relates to individuals, with direct and high-profile policy content, is in the area of immigration and human capital formation. Linkages between individuals and institutions also play an important role in innovation. To what extent do we understand how these linkages to work? What opportunities to understand the role of human capital in innovation within a firm could be gained from human
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
resource surveys and by linking existing data to other data sets? [15 minutes each]
  • – Paula Stephan (Georgia State University; session lead): The global S&E workforce—the “Global Science” Research Project; placement of recent graduates in industry. Data needs, policy issues
  • – Rajshree Agarwal (University of Maryland): Linking knowledge diffusion among firms, industries, and regions to the underlying mechanisms of employee entrepreneurship and mobility
  • – Lee Fleming (UC Berkeley): Patent Data: New Metrics and New Linkages
  • – Jason Owen-Smith (University of Michigan): Networks, physical space, and innovation. Work on IRIS/UMETRICS to collect, improve, protect, and use big data on the dynamics of science and the economy
  • – Alfonso Gambardella (University Bocconi): Managerial practices and incentives for research and innovation; what could be learned from experiments, large and systematic computerized data bases, surveys
Open Discussion [15 minutes]
3:00 Break (refreshments available outside conference Room 120)
3:15 Measuring Public-Sector Innovation and Social Progress
Measuring the full range of benefits generated through innovation requires capturing data on a range of activities that goes beyond those traditionally tracked. Innovation that takes place in the public sector is one area to be explored and is a high-priority topic within the international community. Innovation that advances the public good in ways that take place beyond the market (in noneconomic ways) is another related topic of interest. Both have featured in discussions about revision of the Oslo Manual and in planning for September’s Blue Sky Conference.
Questions to be addressed include the following: What approaches to innovation measurement can be developed for capturing activities within the public sector? Can innovation metrics be developed that bring into scope how the successful exploitation of ideas affects the provision of public goods and the well-being of society more broadly, beyond the contribution to efficiency, effectiveness, or quality in the production of market
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
goods and services? Are there negative social impacts of innovation that should be included in our understanding of the outcomes of innovation? [15 minutes each]
  • – Fred Gault (UNU-MERIT, Maastricht; session lead): Innovation indicators for the public sector. Conceptual overview of what would variables factor into public-sector innovation measurement
  • – Daniel Sarewitz (Arizona State University): Innovation and social progress
  • – Charles Edquist (CIRCLE, Lund University): Design of innovation policy, innovation-related public procurement and a critical assessment of the Innovation Union Scoreboard
  • – Kaye Husbands Fealing (Georgia Tech): Innovation and noneconomic/public health outcomes; GA Tech food security and safety study
  • – Eric von Hippel (MIT): Household-sector innovation
Open Discussion [15 minutes]. Overarching questions for presenters during this session and for discussion afterward:
  • Which aspects of the innovation processes are not well covered and increasing in importance?
  • If we could measure something that is currently not measured, what would it be? Where can funding for measurement be most productively directed?
  • For what aspects of innovation measurement might surveys not be the optimal data collection strategy? Where might administrative or big data become more prominent in meeting demand for new metrics?
  • What is the path forward for measuring innovation (a difficult measure process) in the case of public goods and social progress (a difficult to measure, largely nonmarket outcome)?
5:00 pm Planned Adjournment
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

Friday, May 20 (9 am-2 pm)

9:00 am Welcome Back(breakfast will be available outside conference room 120)
Scott Stern (MIT; workshop chair): Key questions, goals for the day
9:15 Regional Innovation Models and Data Needs
Many of the spillovers and complementarities of innovative activity take place at the local level. What models help policy makers and other data users at the sub-national level? [15 minutes each]
  • – Maryann Feldman (UNC/NSF; session lead): Regional innovation systems
  • – Catherine Fazio (MIT): Measures of innovation quantity and quality—e.g., Entrepreneurial Quality Index, the Regional Entrepreneurship Cohort Potential Index and the Regional Entrepreneurship Acceleration Index—and findings emerging from these measures
  • – Rosemarie Ziedonis (Boston University): Insights from work conducted with Bo Zhao on an R&D loan program in Michigan. Numerous states have launched similar and large-scale programs over the past few decades, yet there are major data roadblocks in obtaining access to the basic pieces of information required to study the effects (e.g., difficulties measuring “innovative activity” of the small companies using standard patent-based measures
  • – Sheryl Winston Smith (Temple University): Measuring micro-foundations of innovation and entrepreneurship, and implications of early funding, intangibles, and peer effects. Leveraging deep, novel data on STEM-based entrepreneurship from earliest stages to construct novel measures of founder, startup, and early hiring characteristics. Findings emerging from research on accelerators and professional angel groups pointing to early influences on growth, hiring, funding, and exit pathways for startups. Implications of geographic and regional measures
  • – Thomas Guevara (Economic Development Administration): Cluster mapping of industry and business environments for use in making strategic
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

investments, recruiting new companies, and laying the groundwork for new industries. [15 minutes each]

Open Discussion [15 minutes]
10:45 Break (refreshments available outside conference room 120)
11:00 Innovation Measurement Agendas of the Future
Going forward, new measures of science and technology describing inputs and outcomes associated with innovative activity will be needed, and multiple data modes will be called upon. Commercial data, some recoverable from Web scraping and other computer science methods, may shape future measurement in a range of areas—e.g., new product introduction, quality change, prices and productivity, product diffusion—where large datasets provide advantage in terms of granularity, timeliness, geographic specificity, etc. To what extent can the digital revolution transform metrics in the area of innovation measurement? What are the roles of specialized surveys in innovative data collection? Administrative records (e.g., linking patent information to other data sources)? [15 minutes each]
  • – Scott Stern (session lead): Innovation that is cumulative vs that which is one off; structural characteristics of innovation. Is it possible to develop statistics that provide clues about application and diffusion of innovation?
  • – Jeff Oldham (Google): Tools beyond surveys instruments that can be used to measure innovation and productivity. Interesting work on patent data for target users
  • – Ron Jarmin (U.S. Census Bureau): Goals/actions/policies that the statistical agency data programs might strive for going forward to measure business dynamism and innovation in high tech industries. Partnerships and collaborations between the agencies
  • – Sallie Keller (Social and Decision Analytics Laboratory): Evaluating nonsurvey data for statistical use; advancing statistical and quantitative data-gathering techniques to enhance research interests
Open Discussion [15 minutes]
12:30 pm Lunch (available outside conference room 120)
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

Synthesis and Directions: Shaping Innovation Data and Indicators for the Future

Scott and Bronwyn will facilitate discussion summarizing guidance to NCSES as it prepares for the OECD Blue Sky III Conference. Insights aimed at broadening and modernizing innovation indicators—through advancement of measurement frameworks, identification of data gaps, and exploitation of new data sources—will be distilled. The goal is to emerge from the meeting envisioning priorities for NCSES data collection programs as the agency seeks to make its statistics relevant to data users and producers across measurement, academic, and practitioner communities spanning both the public and private sectors.
2:00 pm Planned Adjournment
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×

This page intentionally left blank.

Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 113
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 114
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 115
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 116
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 117
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 118
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 119
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 120
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 121
Suggested Citation:"Appendix A: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2017. Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/23640.
×
Page 122
Next: Appendix B: Workshop Participants »
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!