jankowski - findings from an ongoing examination of metrics on innovation
TRANSCRIPT
Findings from an Ongoing Examination of Metrics on Innovation in the
U.S. Business Sector Audrey Kindlon and John E. Jankowski
OECD Blue Sky III Forum Science and Innovation Indicators
“Capturing innovation in firms: do we get it right?”September 19, 2016
National Science FoundationNational Center for Science and Engineering Statistics
www.nsf.gov/statistics/
2
Topics
• Background• Criticisms and Challenges of Existing
Innovation Surveys• Revised Approach to Innovation Measurement• Analysis of Existing Questions (2014)• Differences Between the two Approaches• Next Steps
3
National Science Foundation (NSF) and the National Center for Science and Engineering Statistics
(NCSES)• NSF is an independent federal agency created by Congress in 1950
“to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense…”• NSF has a mandate“...to provide a central clearinghouse for the collection,
interpretation, and analysis of data on scientific and engineering resources and to provide a source of information for policy formulation by other agencies of the Federal Government…”
• This mandate has been fulfilled by NCSES and predecessor agencies
• Further codified in the America Competes Reauthorization Act of 2010 — signed on December 6, 2010 — NCSES was created.
• NCSES is one of the 13 principal Federal statistical agencies in the United States.
4
Official U.S. Business Surveys
• Business R&D and Innovation Survey (BRDIS)– Annual survey (All industries; 5+ employees; N=~44,000; mandatory; RR=71%; R&D
coverage ~85%)– Subject approach for measuring innovation
• Microbusiness, Innovation, Technology and Science (MIST) – Pilot survey (Select industries; 1-9 employees; N=~5,000; voluntary; RR=25% first
phase - 67% second phase)– Object approach for measuring innovation
• Microbusiness R&D and Innovation Survey (MicroBRDIS)– Forthcoming survey (All industries; 1-9 employees with a focus on 1-5; N=~200,000;
mandatory)
5
Background Investigation Steps
• Cognitive interviews with 30 firms in the United States conducted concurrently with 20 firms in Europe
• Interviews with academic researchers specializing in innovation
• A pair of Innovation Metrics Panel Meetings with government leaders /analysts and with academic and business membership organizations
6
Background Findings• Nearly all of the business respondents could readily offer a definition
of innovation • Innovation entailed a commercial achievement and is not the same as
what would be classified as an improvement• Innovation should be successfully commercialized to be considered an
innovation • Few metrics apply consistently to all industries• Few companies maintained formal innovation metrics and records of
innovation • Key challenges of the innovation data currently available are: the time
lag, the fact that industry differences matter, that data sets are often not comparable, and that collecting innovation data is expensive
7
Criticisms and Challenges of Existing Innovation Surveys• Firms don’t maintain centralized repository of
information about new product launches or development of innovative products or services expenses
• International comparability is a challenge because of: • Coverage of industries and size thresholds of companies • The more basic indicators may not be very meaningful, e.g.,
incidence of innovation.• The more meaningful indicators can be complicated and hard to
communicate. • Numerous methodological challenges as well
8
Revised Approach to Innovation Measurement
• MIST was developed to collect national statistics on R&D expenditures and other related statistics, including innovation, among the smallest businesses in the United States.
• MIST questionnaire started with questions similar to BRDIS and Community Innovation Survey
• However, cognitive research showed that respondents would interpret and respond inconsistently.
9
Cognitive Research on Innovation Questions
• Started with the BRDIS questions, followed up by using the BRDIS approach with a definition of product or process innovation.
• Offsetting result, neither of which were “accurate”– Participants seemed to be over-reporting innovation– Participants wanted to report innovations that were only in
development. • Some confusion over the concept of “improving” goods and
services was seen as well. Some participants skipped reading the definition of product innovation.
• For example, some companies that provide software services would interpret and respond that almost everything they create is something new.
10
Revised Approach to Innovation Measurement
• Specific elements of innovation from the Oslo Manual were asked.
• Definitions were incorporated into the question items.
• BRDIS asked only about product and process innovation; MIST had questions on product, process, organizational and marketing innovation
• Each aspect of innovation (product, process, marketing, and organization) was broken into individual yes/no questions.
11
Differences Between the two Approaches
• BRDIS uses the phrase “new or significantly improved goods or services,” MIST does not.
• Questions about new or improved goods that were “new to your market” versus “only new to your company” were deleted.
• MIST asks respondents if their company offered a new good or service that no company ever offered before.
• MIST also asks respondents if their company offered a new good or service that was new only to their company
• Sampled different business sizes (numbers of employees.• MIST was a pilot survey, BRDIS is not.
12
Revised Approach to Innovation MeasurementA product innovation is the market introduction of a new or significantly improved good or service with respect to its capabilities, user friendliness, components, or sub-systems.
• Product innovations (new or improved) must be new to your company, but they do not need to be new to your market.
• Product innovations could have been originally developed by your company or by other companies.
During the three years 2013 to 2015, did your company introduce?
a. New or significantly improved goods (Exclude the simple resale of new goods purchased from other companies and changes of a solely aesthetic nature?
b. New or significantly improved services?
13
Analysis of Existing Questions (2014)
BRDIS: Subject approachMIST: Object approach
14
Innovation Follow up • Experiment conducted by JPSM students at University of Maryland• Respondents who said yes to an innovation were asked to provide
an example • Examples provided were coded and then re-coded to mitigate
against different interpretations• Allows us to link Oslo Manual definitions to actual innovations and
to understand if respondents have full comprehension of what is being asked.
• By asking the follow-up questions, we gain further insight into how close of what respondents think to be innovation adheres to the Oslo Manual guidance on what is innovation.
15
Innovation Follow Up (respondent / inter-coder agreement)
In general, only about one-half to two-thirds of respondents offered a “correct” example of innovation.
16
Innovation Follow Up - Facebook exampleOslo Manual on marketing innovation• The implementation of a marketing method not previously used by the firm. It
must be part of a new marketing concept or strategy that represents a significant departure from the firm’s existing marketing methods. The new marketing method can either be developed by the innovating firm or adopted from other firms or organisations. New marketing methods can be implemented for both new and existing products. (paragraph 171).
This was translated into the following question (question 19b):• Promoted your company’s goods or services in a new way (e.g., the first time
use of a new type of advertising, a new brand image, introduction of loyalty cards).
To which a respondent answered ‘yes’ because the microbusiness had newly used Facebook to advertise.
17
Next Steps• Microbusiness R&D and Innovation Survey
– Based on MIST questionnaire, but with some refinements. Launch in March 2017
• Business Case Studies on Innovation– In depth investigation of existing innovation business
case studies to identify useful generalizations• National Academies’ Board on Science, Technology
and Economic Policy Workshop on Impact of Public Sector Investments on Corporate Innovation
