a theory-based logic model for innovation policy … · a theory-based logic model for innovation...
TRANSCRIPT
A Theory-Based Logic Model for Innovation Policy and Evaluation
Presented atCanadian Evaluation Society Conference
Victoria, British ColumbiaMay 2010
Gretchen Jordan, Sandia National [email protected]
Portions of the work presented here were completed for the U.S. DOE Offices of Science and Energy Efficiency and renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is a multi program laboratory operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed aresolely those of the author.
SAND Number: 2010-2699C
2
• The Challenge– US national interest– Policy evaluation cycle
• Models of what is known about innovation– R&D– Diffusion– Putting these together
• Leverage points for innovation policy– Three levels, multiple institutions– A solar energy example
• Implications for evaluation
Outline
G. Jordan CES May 2010
3
White House S&T Priorities stress evaluation and developing policy tools
Agencies should describe in their budget submission how they are• prioritizing activities toward four challenges and strengthening four
cross-cutting areas (which include productivity of research institutions)
• Expecting outcomes of research in above areas, providing quantitative metrics where possible
• Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs
• Operating in the open innovation model and supporting long term high-risk, high payoff research
Agencies will:• Develop outcome oriented goals for S&T, target investment toward
high performers, develop ‘science of science policy” tools that can improve management and assessment of impact
-Peter Orszag, John Holdren, August 4, 2009 (for the FY 2011 Budget)
G. Jordan CES May 2010
G. Jordan AEA November 2009 4
Evaluation in the policy cycle starts with the rationale for policy
Foresight Technology Roadmapping
TechnologyAssessment
Wolfgang Polt30-10-2007
G. Jordan CES May 2010
G. Jordan AEA November 2009 5
http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant
Currently parts are studied and understood better than the whole!
Most simple view of innovation is linear: R&D followed by adoption of the new product
6
Inputs, Context
• Planning & management, skilled workforce & RTD infrastructure
R&D
• Investigation, new concepts, understanding & research tools• New technology, practice, policy developed• New technology, practice, policy demonstrated
Market Diffusion
• Infrastructure for diffusion of the newly developed technology, practice, or policy
• New technology, practice, policy deployed, adopted
G. Jordan CES May 2010
7
Source: R. Cooper/ Exxon’s Stage Gate
Business Case is
discussed at each
gate
Often-used Stage Gate model explains more about process up to commercial launch
G. Jordan CES May 2010
8
Marketing R&D, Quality R&D
Engineering & manufacturing R&D
6
8
7
Connectivity and Throughput
Production, Refinement
The Idea Innovation Network makes role of manufacturing, quality and
commercialization R&D explicit
G. Jordan CES May 2010
The idea innovation network: Hage and Hollingsworth (2000), modifying Kline and Rosenberg (1986
9
Logic Model 1: Non linear R&D --the Idea Innovation Network theory
• Six R&D arenas• RTD advance can
occur in one or more arenas
• Ideas move between arenas
• Arenas are increasingly differentiated
• Inter-organizational networks transfer tacit knowledge
Basic research
Manufacturingresearch
Applied research
Development research
Quality research
Commercializationresearch
INNOVATION
Universities
Small Tech firms
Largecompanies
. . .
. .
. . . sub networks
An example
Jordan, Hage & Mote, 2007
G. Jordan CES May 2010
The logic of diffusion in a market: a system with four domains
G. Jordan CES May 2010
DOE Impact Evaluation Framework, Reed and Jordan 2007
4 Domains
StrategicPlanning Production Market
DevelopmentValueAdded
EntrepreneurialActivity
RiskReduction
ProprietaryTechnologies
GenericTechnologies
Science Base
Joint Industry-Government Planning
Market Planning Assistance
Acceptance Test Standards; National Test Facilities
Interface Standards
Measurement Standards
National Labs (NIST)
Intellectual Property Rights
National Labs
Direct Funding of Firms, Universities,
Consortia
Technology Transfer (Universities, MEP)
Tax Incentives
Universities
ValueAdded
Targets for Science, Technology, Innovation and Diffusion (STID) Policy
G. Tassey, The Technology Imperative, Edward Elgar, 2007
Tassey’s model stresses technology infrastructure
G. Jordan CES May 2010
Logic Model 2: Diffusion of a technology or practice or policy
12
A system ofFour market domains • End User• Business • Government• Information
Supporting technology infrastructure
Product refinement to get characteristics needed for diffusion (E. Rogers: Relative advantage, Compatibility, Complexity, Trialability, Observability)
G. Jordan CES May 2010
G. Jordan, 2010
Logic Model 3: The Innovation Eco-System: R&D and Market Diffusion
13G. Jordan CES May 2010
Basic research
Manufacturingresearch
Applied research
Development research
Quality research
Commercialization/ Utilizationresearch Information
Infrastructure
Business Infrastructure
TechnologyInfrastructure
ProductRefinement
GovernmentInfrastructure
End UserAttitudes, Action
Interactions
G. Jordan, 2010
Important to consider multiple levels within the system, to focus evaluation on technology sector
G. Jordan AEA November 2009 14
Team/lab
S&T sector
National “rules”
?
• Sectors differ in– Amount of investment by R&D arena– Rates of technical change
• Policy impacts differ by sector
• Mission and policy decisions are often sector specific
Meso/sector level connects macro with micro
bottleneck
15
Important to consider the institutions and actors as well as interactions, infrastructure
Demand
Consumers (final demand)Producers (intermediate demand)
Industrial system Education and research system
Political system
Government
Governance
RTD Policies
Professional educationand training
Higher educationand research
Public sectorresearch
Large companies
Mature small/ mediumenterprises ( SMEs)
New, technology -based firms
Infrastructure
IntermediariesResearchinstitutesBrokers
Banking, venture capital
IPR and information
Innovation andbusiness support
Standardsand norms
Framework conditionsFinancial environment; taxation and incentives; propensity to innovation
and entrepreneurship; mobility
A National Innovation System Model
The potential reachof public policies …
Demand
Consumers (final demand)Producers (intermediate demand)
Industrial system Education and research system
Political system
Government
Governance
RTD Policies
Professional educationand training
Higher educationand research
Public sectorresearch
Large companies
Mature small/ mediumenterprises ( SMEs)
New, technology -based firms
Infrastructure
IntermediariesResearchinstitutesBrokers
Banking, venture capital
IPR and information
Innovation andbusiness support
Standardsand norms
Framework conditionsFinancial environment; taxation and incentives; propensity to innovation
and entrepreneurship; mobility
Source: Arnold and Kuhlman, 2001
A National Innovation System Model
The potential reachof public policies …
G. Jordan CES May 2010
Logic Model 4. The innovation eco-system with leverage points
16G. Jordan CES May 2010
Basic research
Manufacturingresearch
Applied research
Development research
Quality research
Commercialization/ Utilizationresearch Information
Infrastructure
Business Infrastructure
TechnologyInfrastructure
ProductRefinement
GovernmentInfrastructure
End UserAttitudes, Action
Interactions
G. Jordan, 2010
Macro institutions, interactions, infrastructure, actors
Micro institutions, interactions, infrastructure, actors
Meso/Sector
Policy rationale: Examples from DOE’s Solar Program
17G. Jordan CES May 2010
G. Jordan AEA November 2009 18
Conclusion and implications for evaluation
• Innovation occurs within a multi-level, complex, dynamic eco-system.
• Policy rationale, objectives, and evaluation use at least an implicit notion of how the innovation system works.
• Looking at only part of the elephant may give incorrect answers.
• Evaluation using an agreed upon model of the innovation system could – Better test existing theories, and– synthesize theories and build new understanding of
the underlying program theory.